Streaming output corrupts CJK characters at UTF-8 chunk boundaries (displays ���)

Resolved 💬 4 comments Opened Apr 9, 2026 by waterwoodwind Closed May 23, 2026

Description

When Claude Code streams long Chinese (CJK) text output in the terminal, some characters are occasionally replaced with (U+FFFD replacement characters).

Example: 大厂 renders as ���厂

Root Cause Analysis

CJK characters occupy 3 bytes in UTF-8 (e.g., = E5 A4 A7). When the SSE streaming layer splits output into chunks and the chunk boundary falls in the middle of a multi-byte character, the terminal receives incomplete byte sequences and renders them as replacement characters.

Normal:    [...E5 A4 A7...] → 大
Truncated: [...E5] | [A4 A7...] → ���

Steps to Reproduce

  1. Use Claude Code CLI in a terminal (macOS, zsh)
  2. Ask Claude to generate a long response in Chinese (e.g., a multi-section research report with tables)
  3. Observe occasional characters in the streaming output

Expected Behavior

All CJK characters should render correctly during streaming output.

Actual Behavior

Random CJK characters are corrupted to ��� during streaming. The corruption is intermittent and depends on where chunk boundaries fall.

Environment

  • OS: macOS (Darwin 25.3.0, Apple Silicon)
  • Terminal: iTerm2 / Terminal.app / tmux
  • Claude Code: Latest version
  • Shell: zsh

Notes

  • Writing to files (via the Write tool) is NOT affected — the content is correct
  • Only terminal streaming display is affected
  • This appears to be a missing UTF-8 boundary alignment in the streaming output layer
  • The fix would be to buffer incomplete multi-byte sequences at chunk boundaries before flushing to stdout

Suggested Fix

Ensure the streaming output layer aligns chunk boundaries to complete UTF-8 character boundaries. When a chunk ends with an incomplete multi-byte sequence, buffer those trailing bytes and prepend them to the next chunk before writing to stdout.

View original on GitHub ↗

This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗