Streaming output corrupts CJK characters at UTF-8 chunk boundaries (displays ���)

Status Closed — not planned
Maintainer reply None cached
Activity 4 comments · opened Apr 9, 2026 · closed May 23, 2026

Description

When Claude Code streams long Chinese (CJK) text output in the terminal, some characters are occasionally replaced with � (U+FFFD replacement characters).

Example: 大厂 renders as ���厂

Root Cause Analysis

CJK characters occupy 3 bytes in UTF-8 (e.g., 大 = E5 A4 A7). When the SSE streaming layer splits output into chunks and the chunk boundary falls in the middle of a multi-byte character, the terminal receives incomplete byte sequences and renders them as replacement characters.

Normal:    [...E5 A4 A7...] → 大
Truncated: [...E5] | [A4 A7...] → ���

Steps to Reproduce

  1. Use Claude Code CLI in a terminal (macOS, zsh)
  2. Ask Claude to generate a long response in Chinese (e.g., a multi-section research report with tables)
  3. Observe occasional � characters in the streaming output

Expected Behavior

All CJK characters should render correctly during streaming output.

Actual Behavior

Random CJK characters are corrupted to ��� during streaming. The corruption is intermittent and depends on where chunk boundaries fall.

Environment

  • OS: macOS (Darwin 25.3.0, Apple Silicon)
  • Terminal: iTerm2 / Terminal.app / tmux
  • Claude Code: Latest version
  • Shell: zsh

Notes

  • Writing to files (via the Write tool) is NOT affected — the content is correct
  • Only terminal streaming display is affected
  • This appears to be a missing UTF-8 boundary alignment in the streaming output layer
  • The fix would be to buffer incomplete multi-byte sequences at chunk boundaries before flushing to stdout

Suggested Fix

Ensure the streaming output layer aligns chunk boundaries to complete UTF-8 character boundaries. When a chunk ends with an incomplete multi-byte sequence, buffer those trailing bytes and prepend them to the next chunk before writing to stdout.

View original on GitHub ↗

This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗