[BUG] Sonnet 5 writes Korean tool-call parameters as \uXXXX escapes and mis-spells the hex, corrupting Hangul (100% corruption when escape-writing engages; likely root cause of #80009 / #78996)

Summary

We root-caused the frequently-reported Korean (Hangul) corruption in tool-call parameters (AskUserQuestion, TodoWrite — see #80009, #78996): Claude Sonnet 5 sometimes writes Korean tool-parameter strings as \uXXXX unicode escape sequences instead of literal UTF-8, and mis-spells the hex codepoints while doing so. Each mis-spelled escape renders as a valid-but-wrong Hangul syllable — exactly the "syllable substitution producing non-words" symptom in the linked issues.

In a controlled A/B experiment, every single run in which the model engaged escape-writing produced corrupted Hangul (45/45, 100%), at a density of ~3–5% of syllables, while literal-writing runs were clean of this class. A simple output-format instruction fully suppresses it (workaround below).

Root cause evidence

Controlled A/B experiment (2026-07-31)

• Model: claude-sonnet-5 (via Amazon Bedrock global.anthropic.claude-sonnet-5, direct ConverseStream API — no CLI in the loop, raw input_json_delta captured pre-decode; the spontaneous behavior was originally observed in interactive Claude Code sessions on both 1P and Bedrock backends).
• Scenario: forced Korean tool call (AskUserQuestion-style schema: question ≥6 sentences, 3 options with Korean descriptions).
• Arm E: system prompt requires \uXXXX escapes for Korean in tool JSON. Arm L: system prompt forbids escapes (literal UTF-8). n=30/arm, interleaved. Replicated on a second AWS account to rule out account/pool effects.
• Detection: KS X 1001 rare-syllable heuristic + U+FFFD + escape-validity checks on the decoded text (escapes themselves are not counted as corruption), plus an LLM judge for common-syllable substitutions.

| | Account 1 | Account 2 |
|---|---|---|
| Arm E, runs that actually used escapes | 23/23 corrupted (100%) | 22/22 corrupted (100%), incl. 1 invalid-JSON run |
| Arm L (literal) | 1/30 (a distinct issue, see "Residual" below) | 0/30 |
| two-proportion z (E vs L) | +7.01 | +7.21 |

Runs that partially engaged (as few as 6 Hangul escapes) also corrupted. The deciding variable is not the instruction but whether escapes were actually written.

What the mis-spellings look like (real raw output, Arm E)

Raw tool-input JSON text as emitted by the model (single-level escapes), with decodes:

...SSO 로그인 점깔도 완릷된 상황...
→ "SSO 로그인 점깔도 완릷된 상황"   (intended: "SSO 로그인 점검도 완료된 상황")

...작업을 실제로 코딅하기 전에...
→ "작업을 실제로 코딅하기 전에"     (intended: "...코딩하기 전에")

Mis-spelled codepoints observed: 검 U+AC80 → 깔 U+AE54, 료 U+B8CC → 릷 U+B9B7, 딩 U+B529 → 딅 U+B505, plus many landing on non-KS-X-1001 syllables (릷, 딅, 렜, 뱠, …). Writing each syllable as four hand-spelled hex digits gives the model many opportunities to emit a near-miss codepoint; literal UTF-8 writing does not.

Why session logs can't show the fingerprint (note for #80009)

Session JSONL stores tool_use.input as an already-parsed object, so model-written escapes are decoded away before storage — the transcript shows substituted syllables with no trace of the escapes. To capture the fingerprint, use a raw source: claude -p ... --output-format stream-json --include-partial-messages and inspect the partial_json strings (model-written escapes survive there as \\uXXXX in file-raw), or a logging proxy at the wire. Rule of thumb: count backslash layers at the capture point — double backslash in file-raw = the model wrote the escape.

Workaround (verified)

Adding this instruction (memory / CLAUDE.md / system prompt) fully suppressed the corruption class in the field and in the experiment:

Always write Korean (and other non-ASCII) strings in tool-call parameters as literal UTF-8; never as \uXXXX unicode escapes.

Residual, distinct low-rate issue (not fixed by the workaround)

Separately from the escape class, we verified rare literal token-level syllable substitutions (~0.3–0.4% of runs; e.g. 새벽→새벅, 믿고→믄고, 쪽→쪼르) that arrive as clean literal UTF-8 in the raw stream (zero escapes) — the workaround cannot and does not suppress these. We also observed one U+FFFD emission replacing a syllable inside tool-input JSON on the 1P API path (single replacement character with intact continuation, which excludes client-side chunk-split decoding). Happy to file these separately with full raw delta captures if useful.

Environment

• Claude Code CLI 2.1.220 (behavior observed on both 1P/Enterprise and Bedrock backends; reproduced via direct Bedrock API without any CLI, so this is model output behavior, not a client bug)
• Model: claude-sonnet-5
• Korean-language sessions, tool-heavy workflows (AskUserQuestion / TodoWrite)

Suggested fix

Model/decoding-side: avoid hand-spelling \uXXXX hex for CJK in tool-input JSON (emit literal UTF-8), and/or validate/normalize escape sequences in tool-parameter strings at generation time. Full harness code, raw run logs (JSONL with pre-decode deltas), and adversarial verification records are available on request.

Related: #80009, #78996 (symptom reports consistent with this root cause).