[BUG] Streaming output corrupts CJK characters to U+FFFD at UTF-8 chunk boundaries (still repro on 2.1.150)

Summary

Multibyte (CJK) characters in Claude Code's streamed output are intermittently replaced with U+FFFD (REPLACEMENT CHARACTER, �). The same character renders correctly elsewhere in the same response, so this is a non-deterministic chunk-boundary bug — not a font, terminal, or locale issue.

This is the same defect as #45508 ("Streaming output corrupts CJK characters at UTF-8 chunk boundaries") and #40396, with duplicates #42867 / #47013 / #48904. All of those were closed NOT_PLANNED / stale — none were fixed. It still reproduces on the latest CLI (2.1.150). Re-filing with byte-level proof on the current version and a concrete fix to request re-triage.

Environment

• Claude Code: 2.1.150
• OS: macOS (Darwin 24.0.0), Apple Silicon
• Output format: --output-format stream-json
• Language observed: Chinese (Simplified); the same class of corruption was reported for Korean (#47013, #40396) and Japanese (#42867, #48904)

Evidence

Within a single assistant response, the character 台 (U+53F0, UTF-8 E5 8F B0) appeared both correctly and corrupted:

| Occurrence (same response) | Result |
|---|---|
| Windows 平台多个 surface | correct |
| Windows 平台适配 | correct |
| Windows 平���多项适配 | corrupted |

Hex dump of the corrupted run (平…多项适配), taken from the session JSONL log (not the terminal):

e5 b9 b3            平   U+5E73
ef bf bd            �    U+FFFD
ef bf bd            �    U+FFFD
ef bf bd            �    U+FFFD
e5 a4 9a            多   U+591A
e9 a1 b9            项   U+9879
e9 80 82            适   U+9002
e9 85 8d            配   U+914D

One 3-byte character (台 = E5 8F B0) became exactly three U+FFFD. That is the precise signature of a byte split after the first byte (E5 | 8F B0), decoded with non-streaming semantics:

• E5 alone → incomplete lead byte → 1× U+FFFD
• 8F, B0 → two orphaned continuation bytes → 2× U+FFFD

(This matches #47013's Korean hex dump exactly: one 3-byte 습 = EC 8A B5 → 3× U+FFFD.)

Why this is the CLI, not the consumer

The output was captured from the CLI's stream-json stdout via a Python io.TextIOWrapper (incremental UTF-8 decoder), and every line was json.loads()-parsed successfully before being stored. Two consequences:

1. An incremental decoder does not emit U+FFFD for a valid multibyte char split across its own read boundaries — it retains the partial bytes until the next read. So the consumer cannot have introduced these.
2. json.loads() succeeding proves each line was already valid UTF-8 carrying literal U+FFFD code points.

Node always emits valid UTF-8 when writing a JS string to stdout, and --output-format stream-json goes through JSON.stringify. Therefore the U+FFFD was already baked into the CLI's in-memory string before serialization — i.e. the corruption happens in the CLI's SSE-stream → string decode path, upstream of stdout.

Root cause hypothesis

The Anthropic API SSE response is decoded per network/SSE chunk, and a multibyte UTF-8 sequence that straddles a chunk boundary is decoded without carrying the incomplete trailing bytes into the next chunk — e.g. TextDecoder used without { stream: true }, or Buffer.toString('utf8') called on an individual partial chunk. Each orphaned byte then decodes to U+FFFD.

Suggested fix

Decode the SSE byte stream with a streaming decoder that preserves partial multibyte sequences across chunk boundaries:

• const dec = new TextDecoder('utf-8'); dec.decode(chunk, { stream: true }) on every chunk, with a final dec.decode() flush at stream end; or
• accumulate raw Buffers and decode only at complete boundaries (e.g. once per fully-assembled SSE data: event), never on an arbitrary partial chunk.

Reproduction

Intermittent; more likely on longer CJK responses. The per-character corruption probability tracks the chunk-boundary hit rate, so the vast majority of characters are intact and occasional ones corrupt — consistent with every prior report.

Prior reports (all closed without a fix)

• #45508 — Streaming output corrupts CJK characters at UTF-8 chunk boundaries (displays ���) — CLOSED NOT_PLANNED, stale
• #40396 — Korean (CJK) characters corrupted to U+FFFD on macOS + VSCode — CLOSED NOT_PLANNED, stale
• #42867 — U+FFFD in Agent SDK streaming output for CJK text — CLOSED duplicate
• #47013 — Korean characters corrupted during streaming (U+FFFD) — CLOSED duplicate
• #48904 — Japanese (CJK) U+FFFD corruption in streaming response — CLOSED

The recurrence across 5+ reports and multiple languages, combined with closure-by-staleness rather than by fix, is the reason for re-filing on the current version with a byte-level root cause and a concrete patch direction.