Lone UTF-16 surrogate in tool input permanently bricks session (HTTP 400 'no low surrogate')

Status Open

Maintainer reply None cached

Activity 4 comments · opened Jun 14, 2026

Summary

A lone UTF-16 surrogate codepoint emitted by the model inside a tool-call input field permanently disabled a long-running Claude Code session. Every subsequent API turn was rejected with HTTP 400 - The request body is not valid JSON: no low surrogate in string, with the error pointing at the same byte offset on every retry. The session log grew to ~52 MB / 6,545 records and could not be recovered.

Environment

Claude Code CLI on macOS (Darwin 25.5.0), zsh
Korean-language session. BMP Hangul (U+AC00–U+D7A3) lives adjacent to the surrogate block (U+D800–U+DFFF), which appears to make this failure mode more likely.

Reproduction (observed)

During a long session, the model called AskUserQuestion with several Korean options.
One option's description contained \ud99c — a lone high surrogate with no paired low surrogate. Surrounding context was "현\ud99c 보유현황..."; intended text was 현재 (U+C7AC).
The tool call was accepted locally and stored in conversation history.
~33 minutes later, once cumulative request body crossed ~2.2 MB, the Messages API began rejecting every turn with:

``API Error: 400 The request body is not valid JSON: no low surrogate in string: line 1 column 2226149 (char 2226148)``

All 13 subsequent retries failed at the identical offset (char 2226148), confirming the corruption was in conversation history rather than new input. The session became unusable.

Impact

The session is irrecoverable from within Claude Code. /clear discards the full context; retrying re-sends the same poisoned history.
Substantial in-progress work was lost when the session had to be abandoned.
The same byte offset / same error class was hit in a separate earlier session by the same user (~6 hours prior), indicating this is reproducible rather than a one-off.

Root cause (suspected)

A multi-byte Korean codepoint was corrupted into a lone high surrogate somewhere in the model output or tool-call serialization path. Per RFC 8259 §8.2, JSON cannot contain unpaired surrogates, so once one lands in assistant.content[*].input (tool_use), every subsequent request is rejected.

Suggested fixes

Client-side guard (highest leverage): Claude Code should validate tool-call input strings before persisting them to the session transcript. If an unpaired surrogate is detected, either re-prompt the model for that turn or replace the codepoint with U+FFFD and log a warning. This prevents the irrecoverable state.
Server-side recovery path: Expose the offending byte offset in a structured form and allow clients to repair the specific record in-place, rather than requiring the whole session be discarded.
Model-side: Investigate why unpaired surrogates are emitted for Korean text near the BMP surrogate boundary — likely a tokenizer or detokenizer edge case.

Workaround

I have installed a local PreToolUse hook that scans tool inputs for unpaired surrogates and blocks the call before it lands in history. Happy to share the script if useful (~50 lines of Python).

Evidence

Session jsonl: ~/.claude/projects/<project>/f81aa194-c1eb-48e4-902c-7dc4f6bcd83b.jsonl (52 MB, 6,545 records)
First poisoned record: index 6496, timestamp 2026-06-13T07:31:49.622Z, tool_name=AskUserQuestion, codepoint 0xd99c
First 400 error: 2026-06-13T08:05:57.428Z
Last assistant turn before abandonment: 2026-06-13T14:19:20.178Z (13th identical 400)

View original on GitHub ↗

4 Comments

github-actions[bot] · 2 months ago

Found 3 possible duplicate issues:

This issue will be automatically closed as a duplicate in 3 days.

If your issue is a duplicate, please close it and 👍 the existing issue instead
To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

yurukusa · 2 months ago

This is recoverable from outside Claude Code — the work you thought you lost is almost certainly still intact. The poisoned record is sitting in the session's JSONL log on disk, and only one field in it is invalid. Repair that field and the session resumes.
Claude Code persists the whole conversation as a JSONL file (one JSON record per line) under ~/.claude/projects/<project>/<session-id>.jsonl. On every turn it rebuilds the request from that file, which is why the 400 reproduces at the same byte offset forever: the lone \ud99c is stored, not re-generated. Remove that one orphaned surrogate from the file and the request body becomes valid JSON again. Nothing else in the 6,545 records is touched.

Find the file — the ~52 MB one is unmistakable:

``bash ls -laS ~/.claude/projects/*/ | head``

Back it up first (never edit the live file):

``bash cp "<path>.jsonl" "<path>.jsonl.bak"``

Strip only the lone surrogates, preserving valid pairs (emoji, non-BMP CJK). Save as repair.py:

``python import json, re, sys HI, LO = r'[\uD800-\uDBFF]', r'[\uDC00-\uDFFF]' def clean(o): if isinstance(o, str): o = re.sub(HI + r'(?!' + LO + r')', '', o) # lone high surrogate o = re.sub(r'(?<!' + HI + r')' + LO, '', o) # lone low surrogate return o if isinstance(o, list): return [clean(x) for x in o] if isinstance(o, dict): return {k: clean(v) for k, v in o.items()} return o inp, outp = sys.argv[1], sys.argv[2] fixed = 0 with open(outp, 'w', encoding='utf-8') as w: for line in open(inp, encoding='utf-8'): if not line.strip(): w.write(line); continue rec = json.loads(line); c = clean(rec) if c != rec: fixed += 1 w.write(json.dumps(c, ensure_ascii=False) + '\n') print(f"records repaired: {fixed}")``bash python3 repair.py "<path>.jsonl.bak" "<path>.jsonl"``

Resume:

``bash claude --resume <session-id>`json.loads folds valid surrogate **pairs** back into their real codepoint, so they never match the lone-surrogate patterns — emoji and 한글 outside the BMP survive untouched; only the unpaired \ud99c is dropped. I verified on a synthetic log mixing an emoji (😀) with a lone \ud99c: the emoji is preserved, the lone surrogate is removed, every line stays valid JSON and UTF-8-encodable (which is exactly what the Messages API checks). records repaired: 1 is the expected output for your case — it drops a single garbage char from one AskUserQuestion option's description, not any turn, so your context stays whole. If you'd rather not run the script, you can insteadgrep -n 'ud99c' <path>.jsonl to find the one offending line and delete just the orphaned \ud99c` from it by hand — same result.
(Note for the triage bot: the recovery above applies to the linked duplicates #61301 / #61670 / #66932 as well, since they share the stored-lone-surrogate root cause. The real fix is server/client-side surrogate validation at tool-call serialization, but on-disk repair unblocks anyone stuck today.)

Necmttn · 2 months ago

The client-side guard should also write a repair receipt when it intervenes.

Fields I would want: session id, transcript file, record index, byte offset, tool_call_id, offending codepoint class, action taken (reprompt|replace|drop_field|repair_existing), before/after content hash, and resume result. That makes the fix auditable without exposing the full tool input that contained the bad string.

Generated with ax - https://github.com/Necmttn/ax

kcdralph · 1 month ago

Additional repro: same corruption pattern, milder symptom (rendering-only, no session death)

Environment: Claude Code CLI on macOS, running inside cmux (tmux-based multiplexer). Korean-language session.

Confirmed the same corruption class described above, but caught it before it poisoned the conversation history — the session stayed usable, only the rendered text was corrupted.

Isolated repro:

Called AskUserQuestion with plain, correctly-encoded Korean strings (verified by writing them to a file first and diffing — no typo on the caller side):

{"question": "확인", "header": "라벨", "options": [
  {"label": "다", "description": "다"},
  {"label": "아니요", "description": "아니요"}
]}

Rendered in the terminal as:

□ 럼틀        <- header "라벨" corrupted to "럼틀"
황인          <- question "확인" corrupted to "황인"
❯ 1. 다
     다
  2. 아니요
     아니요

Key isolating clue — the preview field is immune:

When the same call also included a preview field (rendered as a separate monospace/markdown box), that field's Korean text came through perfectly intact, even across multiple lines and word-wraps, while question/header/label/description in the same tool call, same screen, same font were corrupted. This rules out terminal/font/locale/wide-character-width causes (which would corrupt everything on screen uniformly) and points specifically at the text path used for question/header/label/description fields — i.e. whatever serializes/re-renders those specific fields is where the surrogate corruption in this issue is being introduced (or surfaced).

This matches the root cause described above: Hangul syllables (U+AC00–U+D7A3) sit adjacent to the surrogate block (U+D800–U+DFFF), and something in the model-output or tool-input handling for these fields is splitting/misencoding a 3-byte UTF-8 Hangul codepoint into an unpaired surrogate, which then renders as a different, unrelated Hangul syllable (not a replacement character — a different valid-looking character, e.g. "확인" → "황인", "라벨" → "럼틀"). That's consistent with a single corrupted byte inside a UTF-8 sequence being reinterpreted as a different valid 3-byte Hangul sequence, rather than a width/wrapping bug.

Given the suggested fix here (validate/guard tool-call input strings before persisting), it'd also be worth checking whether the rendering path for AskUserQuestion's question/header/label/description fields specifically (as opposed to preview, which appears to use a different renderer) shares code with the serialization path that's corrupting the persisted transcript in the OP's report.

Happy to provide the full raw tool-call JSON / screenshots if useful.