›_ Claude Code Issues
Claude Code Issues / #43746

Silent U+FFFD corruption in CJK model output due to TextDecoder missing `{ stream: true }` in SSE line decoder

Status Fixed / completed
Reported on v2.1.92
Maintainer reply None cached
Workaround ✓ Mentioned in thread ↓
Activity 6 comments · opened Apr 5, 2026 · closed Apr 10, 2026

Preflight Checklist

  • [x] I have searched existing issues and this hasn't been reported yet
  • [x] This is a single bug report (please file separate reports for different bugs)
  • [x] I am using the latest version of Claude Code

What's Wrong?

Japanese (CJK) characters in model output are silently replaced with U+FFFD (replacement character) when written to files via Write/Edit tools. No API error occurs and no warning is shown — the corruption is completely silent.

This is not the same issue as #16294 (unpaired surrogates from tool output causing API 400 errors). This bug originates inside the SSE streaming decoder in the Anthropic SDK, not from external tool output.

Example: 銀行送金 becomes U+FFFD U+FFFD U+FFFD 行送金 — the character U+9280 (e9 8a 80, 3-byte UTF-8) is replaced with 3 consecutive U+FFFD.

What Should Happen?

All CJK characters should be correctly decoded from the SSE streaming response and written to files without corruption.

Error Messages/Logs

Steps to Reproduce

  1. Start a Claude Code session with locale C.UTF-8
  2. Ask Claude to write a file containing substantial Japanese text via the Write tool (longer outputs increase the probability of hitting the bug)
  3. Check the written file for U+FFFD:

``bash
LC_ALL=C grep -Pc '\xef\xbf\xbd' <file>
``

  1. The bug is intermittent — it depends on whether an SSE chunk boundary falls in the middle of a multi-byte UTF-8 character

Note: the corruption is also visible in terminal output as garbled characters, and is recorded in the session JSONL transcript.

Claude Model

Opus

Is this a regression?

I don't know

Last Working Version

_No response_

Claude Code Version

2.1.92

Platform

Anthropic API

Operating System

Ubuntu/Debian Linux

Terminal/Shell

WSL (Windows Subsystem for Linux)

Additional Information

Evidence: input is clean, output is corrupted

I analyzed the session JSONL transcript byte-by-byte:

  1. Lines 1-143 contain zero U+FFFD — all input context (CLAUDE.md, rules, settings, every file read via Read tool) is completely clean.
  2. Line 144 (assistant role, Write tool call) is the first U+FFFD occurrence — corruption originates in model output processing, not in input.
  3. All 1,982 text files on disk are clean — no external contamination source.

Corruption pattern

| Pattern | Occurrences | Intended char | UTF-8 bytes |
|---------|------------|--------------|-------------|
| 3 consecutive U+FFFD | 26x | U+9280 | e9 8a 80 |
| 3 consecutive U+FFFD | 19x | U+5206 | e5 88 86 |
| 3 consecutive U+FFFD | 12x | U+30B8 | e3 82 b8 |
| 3 consecutive U+FFFD | 9x | U+5024 | e5 80 a4 |
| 2 consecutive U+FFFD | 3x | U+52D5 | e5 8b 95 |
| 2 consecutive U+FFFD | 2x | U+975E | e9 9d 9e |

Key observation: 3-byte CJK characters produce either 3 or 2 U+FFFD, consistent with byte-level splitting of a multi-byte UTF-8 sequence.

  • 3 U+FFFD: all 3 bytes of the character fell at a chunk boundary and were each decoded as individual invalid bytes
  • 2 U+FFFD: the first byte landed in the previous chunk (1 U+FFFD), the remaining 2 bytes in the next chunk (2 U+FFFD)

Root cause: TextDecoder.decode() without { stream: true }

Reproduced with Node.js TextDecoder:

// Character: U+9280 = e9 8a 80 (3-byte UTF-8)
const chunk1 = Buffer.from([0xe9]);
const chunk2 = Buffer.from([0x8a, 0x80]);

// WITHOUT stream option - each chunk decoded independently
new TextDecoder().decode(chunk1) + new TextDecoder().decode(chunk2)
// Result: "\ufffd\ufffd\ufffd" (3x U+FFFD) <-- REPRODUCES THE BUG

// WITH stream option - incomplete bytes buffered correctly
const d = new TextDecoder();
d.decode(chunk1, { stream: true }) + d.decode(chunk2)
// Result: correct character <-- FIXED

Exact code location in the binary

Claude Code (v2.1.92) is a Bun-compiled single executable (~220 MB). The bug is in the q88 function (Anthropic SDK's SSE LineDecoder), found at binary offsets 107357658 and 216449490:

// q88: converts Uint8Array line to string (minified, 90 bytes)
function q88(H) {
  let $;
  return (JD6 ?? ($ = new globalThis.TextDecoder, JD6 = $.decode.bind($)))(H);
}

This creates a shared TextDecoder instance and caches a bound .decode() without { stream: true }.

Full SSE streaming pipeline trace

fetch().body (ReadableStream<Uint8Array>)
  -> FxH()        wraps with getReader(), yields raw Uint8Array chunks
  -> Zq_()        accumulates bytes, splits by \n\n (SSE record boundary)
                   yields Uint8Array per SSE record
  -> Me.decode()   (LineDecoder) splits by \n at byte level
                   calls q88() per line to convert bytes -> string
  -> GD6.decode()  parses "event:", "data:" SSE fields
  -> JSON.parse()  extracts content_block_delta.delta.partial_json
  -> string concat  accumulates partial_json fragments
  -> JSON.parse()  final tool_use input (e.g., Write tool content)

Zq_ and Me correctly accumulate at the byte level before splitting by newline bytes. However, q88 decodes each split segment with a TextDecoder that has no streaming state. If the byte sequence passed to q88 contains incomplete UTF-8, the incomplete bytes silently become U+FFFD.

Proposed fix (verified, same-length binary patch)

The original q88 is exactly 90 bytes. This same-length replacement adds { stream: true }:

// Before (90 bytes):
function q88(H){let $;return(JD6??($=new globalThis.TextDecoder,JD6=$.decode.bind($)))(H)}

// After (90 bytes):
function q88(H){var D=new TextDecoder;JD6=JD6||(x=>D.decode(x,{stream:!0}));return JD6(H)}

The proper fix should be in the Anthropic SDK source (anthropic-sdk-typescript), in the LineDecoder / SSE decoder module.

Runtime monkey-patching is not possible — BUN_INSPECT_PRELOAD, --preload, and bunfig.toml preload all have no effect on compiled Bun binaries.

Impact on CJK users

This bug disproportionately affects CJK languages (Chinese, Japanese, Korean). ASCII text uses 1-byte characters that virtually never split at a problematic boundary. Japanese text is almost entirely 3-byte UTF-8 sequences, making the probability of hitting a chunk boundary mid-character much higher.

Workaround

PostToolUse hook that detects U+FFFD after every Write/Edit and blocks with an error message:

#!/bin/bash
INPUT=$(cat)
FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
if [ -z "$FILE_PATH" ] || [ ! -f "$FILE_PATH" ]; then exit 0; fi

FFFD_COUNT=$(LC_ALL=C grep -Pc '\xef\xbf\xbd' "$FILE_PATH" 2>/dev/null)
FFFD_COUNT=${FFFD_COUNT:-0}

if [ "$FFFD_COUNT" -gt 0 ]; then
  CONTEXT=$(LC_ALL=C grep -nP '\xef\xbf\xbd' "$FILE_PATH" 2>/dev/null | head -5)
  jq -n --arg file "$FILE_PATH" --arg count "$FFFD_COUNT" --arg context "$CONTEXT" \
    '{ "decision": "block", "reason": "U+FFFD detected in \($file) (\($count) lines). Re-read and fix.\n\($context)" }'
  exit 0
fi
exit 0

Configure in .claude/settings.json:

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Edit|Write",
      "hooks": [{ "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/check-mojibake.sh" }]
    }]
  }
}

This detects corruption after the fact and prompts Claude to re-read and fix, but cannot prevent the corruption itself.

View original on GitHub ↗

6 Comments

github-actions[bot] · 4 months ago

Found 3 possible duplicate issues:

  1. https://github.com/anthropics/claude-code/issues/42867
  2. https://github.com/anthropics/claude-code/issues/40396
  3. https://github.com/anthropics/claude-code/issues/40574

This issue will be automatically closed as a duplicate in 3 days.

  • If your issue is a duplicate, please close it and 👍 the existing issue instead
  • To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

shonnaise · 4 months ago

This issue is related to #42867, #40396, and #40574 but is not a duplicate. Those issues correctly describe the symptom (CJK characters corrupted to U+FFFD at chunk boundaries), but none of them identify the root cause at the code level.

This issue provides:

  1. Byte-level proof that input context is clean — all 143 JSONL lines before the first corruption contain zero U+FFFD, ruling out input contamination
  2. The exact function responsible — q88 in the Anthropic SDK's SSE LineDecoder, which calls TextDecoder.decode() without { stream: true }
  3. Binary offsets in the compiled Bun executable (107357658 and 216449490)
  4. A minimal Node.js reproduction demonstrating that TextDecoder without { stream: true } produces the exact same 3x U+FFFD pattern
  5. A verified same-length binary patch (90 bytes) that fixes the issue by adding { stream: true }
  6. A full trace of the SSE pipeline from fetch().body through FxH -> Zq_ -> Me.decode -> q88 -> JSON.parse

The fix should be applied in the Anthropic SDK source (anthropic-sdk-typescript), in the LineDecoder module where TextDecoder is instantiated. Specifically, the bound .decode() needs { stream: true } to buffer incomplete multi-byte UTF-8 sequences across chunk boundaries.

I believe this issue should remain open as the authoritative root cause analysis, and the other issues can reference it.

shonnaise · 4 months ago

Further analysis: SDK source looks safe in theory, but corruption still occurs

I traced the minified q88 function back to its source in anthropic-sdk-typescript:

src/internal/utils/bytes.ts — decodeUTF8:

let decodeUTF8_: (bytes: Uint8Array) => string;
export function decodeUTF8(bytes: Uint8Array) {
  let decoder;
  return (
    decodeUTF8_ ??
    ((decoder = new (globalThis as any).TextDecoder()), (decodeUTF8_ = decoder.decode.bind(decoder)))
  )(bytes);
}

src/internal/decoders/line.ts — LineDecoder.decode() concatenates incoming bytes into a Uint8Array buffer, splits by newline bytes (0x0a / 0x0d), and calls decodeUTF8 per complete line.

src/core/streaming.ts — iterSSEChunks accumulates bytes and splits by \n\n (SSE record boundary).

Why the SDK source appears safe

The LineDecoder splits on newline bytes (0x0a), which cannot appear inside a multi-byte UTF-8 sequence (continuation bytes are 0x80-0xBF, lead bytes are 0xC0+). So each segment passed to decodeUTF8 should always be a complete UTF-8 sequence, and { stream: true } should be unnecessary.

But corruption still occurs

I have byte-level proof that clean input produces corrupted output in this exact pipeline. Something between the HTTP response body and the JSONL transcript introduces U+FFFD. Possible explanations:

  1. Bun runtime behavior differs from Node.js — Claude Code v2.1.92 is a Bun-compiled binary, not Node.js. Bun's ReadableStream implementation for fetch().body may chunk bytes differently, or its Uint8Array operations may have edge cases not present in Node.js. The SDK is tested against Node.js, not Bun.
  1. Claude Code application layer — There may be a separate code path in Claude Code itself (outside the SDK) that processes streaming text before writing to the JSONL transcript. If that path decodes bytes independently of the SDK's LineDecoder, it could produce U+FFFD.
  1. Edge case in iterSSEChunks — The double-newline detection (Zq_ / iterSSEChunks) scans for \n\n in the accumulated buffer. If the split offset calculation has an off-by-one, it could yield a chunk that ends mid-character. This would only manifest with very specific byte alignments.

Recommendation

Adding { stream: true } to decodeUTF8 in the SDK would be a low-risk defensive fix — if the input is always complete UTF-8 (as the design intends), { stream: true } is a no-op. But if any upstream layer has a subtle chunking bug, it would prevent silent data corruption.

However, the actual root cause may be in the Bun runtime or in Claude Code's application layer rather than in the SDK. I don't have access to Claude Code's source to investigate further.

shonnaise · 4 months ago

Bun runtime findings

Claude Code v2.1.92 is a Bun-compiled binary, not Node.js. Searching the oven-sh/bun repo reveals relevant history of UTF-8 streaming bugs:

  • oven-sh/bun#5542 (closed, fixed Aug 2024): TextDecoder with { stream: true } did not work — multi-byte sequences split across chunks produced incorrect output.
  • oven-sh/bun#3116 (closed, fixed Aug 2023): HTTP client concatenating binary chunks via string addition replaced bytes > 0x80 with U+FFFD.
  • oven-sh/bun#25495 (still open): BOM handling in streaming TextDecoder is still broken when BOM is split across chunks.

Per the spec, Bun's fetch().body ReadableStream can split chunks at arbitrary byte boundaries, including in the middle of multi-byte UTF-8 sequences. This is correct behavior — it is the caller's responsibility to handle this.

SDK source analysis

I also traced the minified q88 back to anthropic-sdk-typescript:

  • src/internal/utils/bytes.ts — decodeUTF8() uses new TextDecoder() without { stream: true }
  • src/internal/decoders/line.ts — LineDecoder splits by newline bytes (0x0a) at the byte level before calling decodeUTF8()
  • src/core/streaming.ts — iterSSEChunks splits by \n\n at the byte level

Since newline bytes cannot appear inside multi-byte UTF-8 sequences, the SDK design is theoretically safe — each segment passed to decodeUTF8 should be complete UTF-8. However, the corruption still occurs in practice.

Possible explanations:

  1. Bun-specific edge case — Given Bun's history of UTF-8 streaming bugs (above), there may be remaining issues in ReadableStream chunking or Uint8Array operations that cause byte misalignment before the LineDecoder sees the data.
  2. Claude Code application layer — A separate code path outside the SDK may process streaming text without proper UTF-8 handling. I don't have access to Claude Code's source to verify.
  3. Off-by-one in iterSSEChunks — The \n\n scan and split offset calculation could yield a chunk ending mid-character under specific byte alignments.

Adding { stream: true } to the SDK's decodeUTF8 would be a low-risk defensive fix — it's a no-op when input is already complete UTF-8, but prevents silent corruption if any upstream layer has a subtle bug.

Hugeria · 4 months ago

Additional data point: corruption also in toolUseResult.stdout on macOS (not only model output)

I hit the same U+FFFD corruption pattern but in a different code path than the model-output/Write-Edit case described above. Sharing byte-level evidence in case it helps narrow down whether this is (a) a single TextDecoder bug shared across paths or (b) multiple sites that each need the same { stream: true } fix.

Environment

  • Claude Code: 2.1.92
  • OS: macOS 26.3.1 (Darwin 25.3.0)
  • Terminal: Ghostty (TERM=xterm-ghostty)
  • Locale: ja_JP.UTF-8
  • Node: v24.1.0
  • CLAUDE_CODE_NO_FLICKER=1 enabled (ruled out as cause — see below)

Symptom

Single U+FFFD bytes (ef bf bd) persisted in session .jsonl files, primarily inside toolUseResult.stdout — i.e. corruption of child-process stdout captured by the Bash tool, not model streaming output written via Write/Edit.

Example from a session transcript (JSON path .toolUseResult.stdout):

..."benefit": "1�...

And from a rendered table the model generated in a previous session:

│      観点      │      状��態      │
│ データ供給     │ 部分稼働       │
│ スクレイパー群 │ 稼働中だが脆弱 │

Note the single FFFD mid-character in 状��態 — consistent with one byte of a 3-byte UTF-8 sequence landing in a previous chunk and being decoded independently, then the remaining bytes in the next chunk also being mis-decoded. Same signature as #43746's analysis.

Timeline (rules out local renderer)

Scanning all .jsonl in ~/.claude/projects/<project>/ for ef bf bd:

2026-03-15   1 FFFD-lines
2026-03-29   1
2026-03-30   1
2026-03-31   4 (2 files)
2026-04-01   8 (3 files)
2026-04-05   6 (5 files)
2026-04-07   3

14 sessions / ~30 FFFD-containing lines over 3+ weeks.

I initially suspected CLAUDE_CODE_NO_FLICKER=1 (the experimental renderer, enabled 2026-04-05), but the earliest corruption predates it by 3 weeks (2026-03-15). The renderer writes nothing to .jsonl — the corruption is upstream of display.

What I think this adds to #43746

@shonnaise's analysis focuses on the SSE LineDecoder → decodeUTF8 path for model output. My corruption is in toolUseResult.stdout, which is the captured stdout of a child process spawned by the Bash tool. This is a separate code path that wouldn't go through the Anthropic SDK's SSE decoder at all — it reads directly from the child process's stdout pipe.

If toolUseResult.stdout is also exhibiting the exact same FFFD-at-chunk-boundary pattern, then either:

  1. Claude Code has a second TextDecoder without { stream: true } in its subprocess-stdout-capture path (separate fix needed), or
  2. Both paths funnel through a shared decoder utility that's missing the option (single fix covers both), or
  3. Bun's fetch().body / subprocess.stdout ReadableStream has the chunking edge case mentioned in the earlier comments, affecting every consumer equally.

Repro trigger (probabilistic)

The corruption is more likely when Bash tool runs a command producing large Japanese stdout (e.g., find over a path with Japanese filenames, grep/cat of large Japanese-containing files, tool reports that pretty-print CJK tables). Short stdout almost never hits it. This matches the "chunk boundary lottery" hypothesis.

Workaround

Same PostToolUse hook as in the original issue, but matched against Bash as well — detects FFFD in command output and re-runs. Does not prevent corruption, only flags it.

Happy to provide a sanitized excerpt of a JSONL transcript containing the toolUseResult.stdout corruption if useful for triage.

github-actions[bot] · 4 months ago

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.

Related Claude Code issues

Matched on shared title terms, labels, topic and reported release — ranked so issues with real discussion come first.

Browse by topic

This issue appears in the ranked issue list on this topic page, alongside a backlog trend for the whole topic.

Windows & WSL problems →

Browse all Claude Code issue topics →