Silent U+FFFD corruption in CJK model output due to TextDecoder missing `{ stream: true }` in SSE line decoder

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

Japanese (CJK) characters in model output are silently replaced with U+FFFD (replacement character) when written to files via Write/Edit tools. No API error occurs and no warning is shown — the corruption is completely silent.

This is not the same issue as #16294 (unpaired surrogates from tool output causing API 400 errors). This bug originates inside the SSE streaming decoder in the Anthropic SDK, not from external tool output.

Example: 銀行送金 becomes U+FFFD U+FFFD U+FFFD 行送金 — the character U+9280 (e9 8a 80, 3-byte UTF-8) is replaced with 3 consecutive U+FFFD.

What Should Happen?

All CJK characters should be correctly decoded from the SSE streaming response and written to files without corruption.

Error Messages/Logs

Steps to Reproduce

1. Start a Claude Code session with locale C.UTF-8
2. Ask Claude to write a file containing substantial Japanese text via the Write tool (longer outputs increase the probability of hitting the bug)
3. Check the written file for U+FFFD:

``bash
LC_ALL=C grep -Pc '\xef\xbf\xbd' <file>
``

1. The bug is intermittent — it depends on whether an SSE chunk boundary falls in the middle of a multi-byte UTF-8 character

Note: the corruption is also visible in terminal output as garbled characters, and is recorded in the session JSONL transcript.

Claude Model

Opus

Is this a regression?

I don't know

Last Working Version

_No response_

Claude Code Version

2.1.92

Platform

Anthropic API

Operating System

Ubuntu/Debian Linux

Terminal/Shell

WSL (Windows Subsystem for Linux)

Additional Information

Evidence: input is clean, output is corrupted

I analyzed the session JSONL transcript byte-by-byte:

1. Lines 1-143 contain zero U+FFFD — all input context (CLAUDE.md, rules, settings, every file read via Read tool) is completely clean.
2. Line 144 (assistant role, Write tool call) is the first U+FFFD occurrence — corruption originates in model output processing, not in input.
3. All 1,982 text files on disk are clean — no external contamination source.

Corruption pattern

| Pattern | Occurrences | Intended char | UTF-8 bytes |
|---------|------------|--------------|-------------|
| 3 consecutive U+FFFD | 26x | U+9280 | e9 8a 80 |
| 3 consecutive U+FFFD | 19x | U+5206 | e5 88 86 |
| 3 consecutive U+FFFD | 12x | U+30B8 | e3 82 b8 |
| 3 consecutive U+FFFD | 9x | U+5024 | e5 80 a4 |
| 2 consecutive U+FFFD | 3x | U+52D5 | e5 8b 95 |
| 2 consecutive U+FFFD | 2x | U+975E | e9 9d 9e |

Key observation: 3-byte CJK characters produce either 3 or 2 U+FFFD, consistent with byte-level splitting of a multi-byte UTF-8 sequence.

• 3 U+FFFD: all 3 bytes of the character fell at a chunk boundary and were each decoded as individual invalid bytes
• 2 U+FFFD: the first byte landed in the previous chunk (1 U+FFFD), the remaining 2 bytes in the next chunk (2 U+FFFD)

Root cause: TextDecoder.decode() without { stream: true }

Reproduced with Node.js TextDecoder:

// Character: U+9280 = e9 8a 80 (3-byte UTF-8)
const chunk1 = Buffer.from([0xe9]);
const chunk2 = Buffer.from([0x8a, 0x80]);

// WITHOUT stream option - each chunk decoded independently
new TextDecoder().decode(chunk1) + new TextDecoder().decode(chunk2)
// Result: "\ufffd\ufffd\ufffd" (3x U+FFFD) <-- REPRODUCES THE BUG

// WITH stream option - incomplete bytes buffered correctly
const d = new TextDecoder();
d.decode(chunk1, { stream: true }) + d.decode(chunk2)
// Result: correct character <-- FIXED

Exact code location in the binary

Claude Code (v2.1.92) is a Bun-compiled single executable (~220 MB). The bug is in the q88 function (Anthropic SDK's SSE LineDecoder), found at binary offsets 107357658 and 216449490:

// q88: converts Uint8Array line to string (minified, 90 bytes)
function q88(H) {
  let $;
  return (JD6 ?? ($ = new globalThis.TextDecoder, JD6 = $.decode.bind($)))(H);
}

This creates a shared TextDecoder instance and caches a bound .decode() without { stream: true }.

Full SSE streaming pipeline trace

fetch().body (ReadableStream<Uint8Array>)
  -> FxH()        wraps with getReader(), yields raw Uint8Array chunks
  -> Zq_()        accumulates bytes, splits by \n\n (SSE record boundary)
                   yields Uint8Array per SSE record
  -> Me.decode()   (LineDecoder) splits by \n at byte level
                   calls q88() per line to convert bytes -> string
  -> GD6.decode()  parses "event:", "data:" SSE fields
  -> JSON.parse()  extracts content_block_delta.delta.partial_json
  -> string concat  accumulates partial_json fragments
  -> JSON.parse()  final tool_use input (e.g., Write tool content)

Zq_ and Me correctly accumulate at the byte level before splitting by newline bytes. However, q88 decodes each split segment with a TextDecoder that has no streaming state. If the byte sequence passed to q88 contains incomplete UTF-8, the incomplete bytes silently become U+FFFD.

Proposed fix (verified, same-length binary patch)

The original q88 is exactly 90 bytes. This same-length replacement adds { stream: true }:

// Before (90 bytes):
function q88(H){let $;return(JD6??($=new globalThis.TextDecoder,JD6=$.decode.bind($)))(H)}

// After (90 bytes):
function q88(H){var D=new TextDecoder;JD6=JD6||(x=>D.decode(x,{stream:!0}));return JD6(H)}

The proper fix should be in the Anthropic SDK source (anthropic-sdk-typescript), in the LineDecoder / SSE decoder module.

Runtime monkey-patching is not possible — BUN_INSPECT_PRELOAD, --preload, and bunfig.toml preload all have no effect on compiled Bun binaries.

Impact on CJK users

This bug disproportionately affects CJK languages (Chinese, Japanese, Korean). ASCII text uses 1-byte characters that virtually never split at a problematic boundary. Japanese text is almost entirely 3-byte UTF-8 sequences, making the probability of hitting a chunk boundary mid-character much higher.

Workaround

PostToolUse hook that detects U+FFFD after every Write/Edit and blocks with an error message:

#!/bin/bash
INPUT=$(cat)
FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
if [ -z "$FILE_PATH" ] || [ ! -f "$FILE_PATH" ]; then exit 0; fi

FFFD_COUNT=$(LC_ALL=C grep -Pc '\xef\xbf\xbd' "$FILE_PATH" 2>/dev/null)
FFFD_COUNT=${FFFD_COUNT:-0}

if [ "$FFFD_COUNT" -gt 0 ]; then
  CONTEXT=$(LC_ALL=C grep -nP '\xef\xbf\xbd' "$FILE_PATH" 2>/dev/null | head -5)
  jq -n --arg file "$FILE_PATH" --arg count "$FFFD_COUNT" --arg context "$CONTEXT" \
    '{ "decision": "block", "reason": "U+FFFD detected in \($file) (\($count) lines). Re-read and fix.\n\($context)" }'
  exit 0
fi
exit 0

Configure in .claude/settings.json:

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Edit|Write",
      "hooks": [{ "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/check-mojibake.sh" }]
    }]
  }
}

This detects corruption after the fact and prompts Claude to re-read and fix, but cannot prevent the corruption itself.