Streaming output corrupts CJK characters at UTF-8 chunk boundaries (displays ���)
Description
When Claude Code streams long Chinese (CJK) text output in the terminal, some characters are occasionally replaced with � (U+FFFD replacement characters).
Example: 大厂 renders as ���厂
Root Cause Analysis
CJK characters occupy 3 bytes in UTF-8 (e.g., 大 = E5 A4 A7). When the SSE streaming layer splits output into chunks and the chunk boundary falls in the middle of a multi-byte character, the terminal receives incomplete byte sequences and renders them as replacement characters.
Normal: [...E5 A4 A7...] → 大
Truncated: [...E5] | [A4 A7...] → ���
Steps to Reproduce
- Use Claude Code CLI in a terminal (macOS, zsh)
- Ask Claude to generate a long response in Chinese (e.g., a multi-section research report with tables)
- Observe occasional
�characters in the streaming output
Expected Behavior
All CJK characters should render correctly during streaming output.
Actual Behavior
Random CJK characters are corrupted to ��� during streaming. The corruption is intermittent and depends on where chunk boundaries fall.
Environment
- OS: macOS (Darwin 25.3.0, Apple Silicon)
- Terminal: iTerm2 / Terminal.app / tmux
- Claude Code: Latest version
- Shell: zsh
Notes
- Writing to files (via the Write tool) is NOT affected — the content is correct
- Only terminal streaming display is affected
- This appears to be a missing UTF-8 boundary alignment in the streaming output layer
- The fix would be to buffer incomplete multi-byte sequences at chunk boundaries before flushing to stdout
Suggested Fix
Ensure the streaming output layer aligns chunk boundaries to complete UTF-8 character boundaries. When a chunk ends with an incomplete multi-byte sequence, buffer those trailing bytes and prepend them to the next chunk before writing to stdout.
This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗