[BUG] API Error 400 "no low surrogate in string" when Bash output contains invalid Unicode

Found 3 possible duplicate issues:

1. https://github.com/anthropics/claude-code/issues/1709
2. https://github.com/anthropics/claude-code/issues/13157
3. https://github.com/anthropics/claude-code/issues/15027

This issue will be automatically closed as a duplicate in 3 days.

• If your issue is a duplicate, please close it and 👍 the existing issue instead
• To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

Follow-up: Hook-based workaround is not possible

I attempted to implement a workaround using Claude Code hooks, but discovered an architectural limitation that makes this impossible to fix at the hook level.

What I Tried

Added a sanitize_unicode() function to my PostToolUse hook (output_validator.py) that:

• Detects unpaired surrogates via text.encode("utf-8")
• Replaces invalid characters with errors="replace"

Why It Doesn't Work

PostToolUse hooks cannot modify tool_result.

| Hook Type | Available Output Fields | Can Modify Output? |
|-----------|------------------------|-------------------|
| PreToolUse | updatedInput, permissionDecision | Yes (command input) |
| PostToolUse | additionalContext, systemMessage | No (read-only) |

The hook schema explicitly defines:

• PreToolUse: Has updatedInput to modify the command before execution
• PostToolUse: Only has additionalContext to add commentary—no updatedResult equivalent

My sanitization runs, but the original corrupted tool_result is still serialized and sent to the API. The hook's sanitized copy is discarded.

Confirmed Root Cause Location

The sanitization must happen in Claude Code core, specifically:

1. After the Bash tool captures stdout/stderr
2. Before the tool_result is added to the conversation context

There is no hook point between these two steps that allows modification.

Recommendation

Add Unicode sanitization in the Bash tool's output processing, similar to how the tool already has size limiting (30KB BASH_MAX_OUTPUT_LENGTH):

// Pseudocode for where the fix should go
function processBashOutput(output) {
  const truncated = truncateToMaxLength(output);  // Existing
  const sanitized = sanitizeUnicode(truncated);   // NEW
  return sanitized;
}

This would catch invalid Unicode from any source (mutmut, binary data, corrupted files) before it can break JSON serialization.

GitHub Issue #16294 Follow-up: Workaround Implemented

Issue: Invalid Unicode in Bash Output Causes API Errors

Status: Workaround implemented until Anthropic fixes the core issue

---

The Problem

Running mutmut (mutation testing tool) via Claude Code crashes with:

API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"The request body is not valid JSON: no low surrogate in string..."}}

Root cause: mutmut's terminal output contains invalid Unicode characters (unpaired UTF-16 surrogates from ANSI escape sequences and progress bars). When Claude Code serializes the bash output to JSON for the API, these invalid surrogates cause JSON encoding to fail.

Why hooks don't help: PostToolUse hooks can only add additionalContext—they cannot modify the original tool_result. The corrupted output is still serialized and sent to the API.

---

The Workaround

Created scripts/mutmut_safe.sh — a wrapper script that intercepts mutmut output before Claude sees it.

How It Works

1. Output Redirection: Instead of letting mutmut write directly to stdout (which Claude captures), the wrapper redirects all output to a temp file (var/logs/mutmut-output.log)

1. Sanitization Pipeline: After mutmut completes, the output is piped through sanitization filters:

• Strip ANSI escape sequences (\x1b[...m patterns)
• Remove all non-printable/control characters
• Limit output size to prevent token bloat

1. Safe Return: Only the sanitized output is displayed to stdout, which Claude then captures. The corrupted bytes never reach Claude's API serialization.

Files

| File | Purpose |
|------|---------|
| scripts/mutmut_safe.sh | Wrapper script |
| var/logs/mutmut-output.log | Raw output (may contain invalid chars) |
| var/logs/mutmut-safe.log | Sanitized output (safe for Claude) |

Usage

# Safe commands (via Claude Code)
make mutation              # Run mutation tests
make mutation-results      # Show results
make mutation-show ID=123  # Show specific mutant

# Direct mutmut (separate terminal only, NOT via Claude)
make mutation-unsafe

---

Why This Works

The key insight is that Claude Code's crash happens during JSON serialization of tool output, not during command execution. By ensuring only valid UTF-8 reaches stdout, we prevent the serialization failure entirely.

The wrapper acts as a "Unicode firewall" between mutmut and Claude Code.

---

Proper Fix (for Anthropic)

The correct solution is for Claude Code to sanitize bash output before JSON serialization, similar to how it already handles output size limits (30KB BASH_MAX_OUTPUT_LENGTH). A simple regex replacement of unpaired surrogates with the Unicode replacement character (U+FFFD) would prevent this class of errors entirely.

Until that fix ships, this wrapper provides a reliable workaround for mutation testing workflows.

Additional data point and workaround

We encountered this same error while using Claude Code programmatically via @link-assistant/hive-mind (an AI issue solver that orchestrates Claude Code sessions).

Our case

• Claude Code version: Latest (via claude CLI)
• Model: claude-opus-4-5-20251101
• Context: Claude was analyzing Rust source code and log files from a previous solve session
• Error: "no low surrogate in string: line 1 column 160309 (char 160308)"
• Session: 21 turns, ~$1.03 spent before the crash
• Full log: solution-draft-log-pr-1769796128836.txt

Root cause analysis

The issue is that JSON.stringify() in JavaScript does NOT reject lone surrogates — it produces technically invalid JSON that JavaScript's own JSON.parse() accepts (lenient parser), but Anthropic's server-side parser (likely Rust/Python) correctly rejects per RFC 8259 Section 8.1.

Common sources of lone surrogates:

1. Binary file content interpreted as text (tool reads of compiled output, .af files, etc.)
2. Log files with mixed encodings
3. Terminal escape sequences from mutation testing tools, progress bars, etc.

Suggested fix

Add a surrogate sanitization step in the tool result processing pipeline, before the result is added to the conversation history. Here's a minimal implementation:

const sanitizeSurrogates = str => {
  if (typeof str !== 'string') return str;
  return str.replace(
    /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g,
    '\uFFFD'
  );
};

This regex:

• Matches high surrogates (U+D800-U+DBFF) NOT followed by a low surrogate
• Matches low surrogates (U+DC00-U+DFFF) NOT preceded by a high surrogate
• Replaces them with U+FFFD (Unicode replacement character)
• Preserves valid surrogate pairs (used for emoji and supplementary characters)

Our workaround

We implemented this sanitization in our wrapper (hive-mind) at two points:

1. Prompts sent to the Claude CLI are sanitized before being passed
2. safeJsonStringify() in our interactive mode sanitizes string values during JSON serialization

This doesn't fix the core issue (tool output corruption within Claude Code's internal conversation history), but it prevents the prompts we control from triggering the error.

Detailed case study: https://github.com/link-assistant/hive-mind/blob/main/docs/case-studies/issue-1204/README.md
Related issue: https://github.com/link-assistant/hive-mind/issues/1204

Can confirm this is still an issue - especially when using Playwright MCP to get console logs and snapshots. Might try to use this sanitize function. Or just strip emojis out of all our console logs. Who knew that emojis are Human-readable but Not Claude-friendly. 🤷🏻‍♂️

Additional data point: <persisted-output> truncation creates orphaned surrogates

We encountered this error in a slightly different scenario than Bash output - the truncation feature itself is a source of the problem.

Our case

• Claude Code version: 2.1.41
• Model: claude-opus-4-5-20251101
• Context: Claude was fetching GitHub issue comments containing emojis (🤖, 🚀, etc.)
• Trigger: The tool output was 44.8KB, triggering the <persisted-output> truncation to ~2KB preview
• Error position: Column 28642 - exactly where the truncation cut through an emoji surrogate pair
• Full log: solution-draft-log-pr-1771359469877.txt

Root cause identified

The truncation in <persisted-output> is byte/character-based, not Unicode-aware. When the truncation point lands in the middle of a UTF-16 surrogate pair (which emojis require), it leaves an orphaned high surrogate like \ud83e.

Evidence from the log file (position ~98553):

All changes have been merged to the main branch.\n\n---\n\ud83e\n...\n</persisted-output>

The \ud83e is the orphaned high surrogate from what was originally likely 🤖 (U+1F916 = \ud83e\udd16).

Impact

This makes the error more unpredictable than terminal escape sequences - any emoji-containing GitHub comment, PR description, or issue body can trigger it when the output happens to exceed the truncation threshold and the cut point lands on an emoji.

Suggested fix

The truncation logic needs to be Unicode-aware:

function unicodeAwareTruncate(text, maxChars) {
  // Use spread operator to iterate code points, not UTF-16 code units
  const chars = [...text];
  if (chars.length <= maxChars) return text;
  return chars.slice(0, maxChars).join('');
}

Or apply the surrogate sanitization after truncation but before JSON serialization.

Related case study

Full analysis: link-assistant/hive-mind case study

Additional reproduction: Windows + MCP tool output + context compaction

Environment

• OS: Windows 11 Home 10.0.26200
• CLI: Claude Code v2.1.51
• Model: claude-opus-4-6

Reproduction steps

1. Start a session in a large project (~2,700 source files, Korean + English content)
2. Run an Explore subagent (29 tool uses, ~83K tokens output)
3. Read 7 files + pattern search
4. Run Playwright MCP tool (browser_navigate → browser_network_requests → browser_console_messages)
5. At this point, conversation context reaches ~190K characters
6. Next API call fails with: API Error: 400 "no low surrogate in string: line 1 column 190443"

Key observation: Not caused by file content

I scanned all 2,889 source files + 331 config/memory/skill/command/agent files for lone surrogates — zero issues found. All files pass JSON.stringify() → JSON.parse() round-trip.

This strongly suggests the surrogate split occurs during Claude Code's internal context assembly or compaction, not from file content. Specifically:

• The error position (~190K) coincides with where context compaction would truncate
• Large subagent outputs (83K tokens) combined with MCP tool results create the conditions
• Once the error occurs, the session is permanently broken — every subsequent API call includes the corrupted context

Additional finding: MEMORY.md auto-load file

In a separate instance, a broken emoji (📁 stored as lone surrogate 0xD83D without 0xDCC1) in an auto-loaded MEMORY.md file caused the same error on every session start. This was fixed by replacing the emoji with plain text. However, it's unclear how the emoji became corrupted in the first place — it may have been corrupted during a previous context compaction/write cycle.

Suggested fix

1. Sanitize all tool outputs — replace lone surrogates with U+FFFD before including in API requests
2. Sanitize during context compaction — ensure string truncation never splits a surrogate pair
3. Sanitize auto-loaded files — validate MEMORY.md, CLAUDE.md on session start

Additional failure mode: the bug can also break compaction, not just the immediately following API call.

In our case, a GitHub CI API response contained a job name with an emoji truncated mid-surrogate pair (\ud83e without its following \ude73). This got stored in the conversation JSONL. The session continued working for a while — the invalid Unicode was dormant in the history. Later, a normal prompt failed with the surrogate error. Attempting manual compaction to recover also failed with the same error:

Error: Error during compaction: Error: API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"The request body is not valid JSON: no low surrogate in string: ..."}}

This makes the bug harder to diagnose: the failure doesn't necessarily happen on the tool call that introduced the bad Unicode — it can surface much later, on an unrelated prompt, once the conversation grows large enough that the offending content appears in the request body. By then the connection to the original tool output is not obvious.

Workaround for permanently broken sessions:

If your session is stuck in a compaction loop, you can recover it by editing the JSONL file directly:

1. Find the session file: ~/.claude/projects/<project-slug>/<session-uuid>.jsonl
2. Back it up: cp session.jsonl session.jsonl.bak
3. Replace lone surrogates with the replacement character:

import re

path = 'path/to/session.jsonl'
with open(path, 'r', errors='replace') as f:
    content = f.read()

high_surr = re.compile(r'\\u([dD][89aAbB][0-9a-fA-F]{2})')
low_surr_pat = re.compile(r'\\u[dD][cCdDeEfF][0-9a-fA-F]{2}')

result = []
i = 0
while i < len(content):
    m = high_surr.search(content, i)
    if m is None:
        result.append(content[i:])
        break
    result.append(content[i:m.start()])
    after = content[m.end():m.end()+6]
    if not low_surr_pat.fullmatch(after):
        result.append(r'\ufffd')  # replace lone high surrogate
    else:
        result.append(m.group(0))
    i = m.end()

with open(path, 'w') as f:
    f.write(''.join(result))

1. Reopen the session — compaction should succeed now.

Additional reproduction case: mutmut on Linux

Same issue. Running a mutmut script via Bash tool that outputs per-mutant status lines (100+ lines with killed/survived/timeout results). Session becomes permanently stuck after the output is appended to context.

Context:

• Platform: Linux (Ubuntu, kernel 6.8.0)
• Claude Code: 2.1.89
• Tool call that broke the session: Bash(sleep 100 && ps -p 306996 -o etime --no-headers 2>/dev/null || echo "finished" && tail -3 /tmp/mutmut_rerun.log)
• Error: no low surrogate in string: line 1 column 269049 (char 269048)
• mutmut output likely contains ANSI escape sequences, raw diff fragments, and progress bar characters

No recovery possible — /compact, new input, Ctrl+C all produce the same 400 error. Only fix is killing the session (losing 120+ min of context).

Would really benefit from output sanitization before appending to conversation history.

Still unresolved

Bumping this — still hitting the 400 error when Bash output contains invalid Unicode surrogates. The fix (Unicode sanitization) was described in the issue, would be great to see it addressed.

<img width="895" height="155" alt="Image" src="https://github.com/user-attachments/assets/79c8254e-ca4d-4c43-8e40-57a54a8762be" />

Additional failure mode: desktop app's Code tab — every request fails at fixed position, regardless of input

Adding a new data point that significantly widens the scope of this bug. In the macOS Claude desktop app (Code tab), every request to the model 400s with invalid high surrogate in string at a fixed position, even on a brand-new session with the most
trivial possible prompt. The desktop Code surface is effectively non-functional for me on macOS.

Environment

• OS: macOS (Darwin 25.4.0)
• Surface: Claude desktop app → "Code" tab
• Mode: Local (the default; mode picker confirms Local is selected)
• Model: Opus 4.7 1M
• Workspaces tested: two separate team-plan workspaces under the same account — same error in both
• Local config: heavily customized ~/.claude/ (custom subagents, skills, MCP servers, hooks) — but the bug is independent of this, see below

Reproduction

1. Open the Claude desktop app, switch to the Code tab
2. Click "+ New session", pick Local mode, point it at any folder (I tested home dir and several project dirs)
3. Type literally any prompt and submit. I tested:

• ls ~/.claude/agents/
• a natural-language invocation of a custom subagent
• so, what can you help me with? ← bare conversational, no tools, no files, no MCP

Every request returns:
API Error: 400 {"type":"error","error":{"type":"invalid_request_error",
"message":"The request body is not valid JSON: invalid high surrogate in string: line 1 column 1511 (char 1510)"}}

Evidence the bug is in fixed app-injected content (not user data)

The error position is identical across every request:

• Different prompts → same column 1511 char 1510
• Different workspaces → same position
• Different sessions, different folders → same position
• Different request_ids (so requests are reaching the API, not cached): req_011CajZXjJxFaVWUfufno4sw, req_011CajZYrJfajt8fogrPVh3i, req_011CajZyEWZyHn4CAQGAWyEG, req_011CajoJLG98FNq63C7spCHk

I scanned ~/.claude/ (agents/skills/memory/settings/hooks) for non-BMP characters; the only astral chars were five 🟢 Live status lines in agent files. I replaced them with [LIVE] to eliminate them. The error position did not move by a single char.
That rules out user config as the source — the bad character is in something the desktop app injects into the request body before any user content.

This is consistent with @konard's earlier observation that <persisted-output> truncation can produce orphaned surrogates: the fact that char 1510 stays nailed across totally unrelated inputs strongly suggests a fixed-length truncation in the desktop
app's request prefix landing inside a UTF-16 surrogate pair.

Why this matters more than the existing reports

Most prior reports in this thread (and the closed duplicates #1709, #5761, #5934, #21095) describe accumulated history corruption — a Bash/Playwright/MCP tool returns bad Unicode and corrupts subsequent calls. Workarounds exist: start a new session,
sanitize console logs, etc.

This variant has no such workaround, because:

• Fresh sessions fail
• Trivial first prompts fail
• No MCP, no tools, no /agents, no files involved
• Local config doesn't matter

The entire desktop Code surface on macOS is non-functional. As a separate symptom, /agents returns a hardcoded "/agents isn't available in this environment" regardless — the surface clearly isn't a fully-fledged Claude Code instance.

Status of this bug class

For context for anyone arriving here: this bug class has been reported continuously since June 2025 (#1709), with multiple detailed reproductions, root-cause analysis (#16294 by @coygeek), and a one-line sanitization fix already proposed (#21095).
The pattern across reports is that the issue auto-closes for inactivity and gets refiled. Eight months in, on a paid product, with the proposed fix being roughly five lines of regex — this would benefit from someone on the team picking it up.

The desktop app variant pushes the impact wider: it's no longer a "long sessions occasionally break" bug, it's a "this entire product surface ships broken" bug.

I also have this issue and it keeps killing sessions of mine. Awful UX, please fix ASAP.

+1, hitting this on Windows 11 zh-CN (codepage 936 / GBK) with Claude Code 2.x, and I want to add a data point because every prior report in this thread is macOS/Linux or blames the tool's stdout for being invalid.
My case proves the corruption can originate inside Claude Code's own subprocess decoding layer, with the underlying tool emitting perfectly valid UTF-8.

## Origin mode: valid UTF-8 in → lone surrogate out, via cp936 + surrogateescape

Source character: — (U+2014, em dash). On the wire it's the standard 3-byte UTF-8 sequence E2 80 94. Nothing wrong with the input.

A Python child process spawned by the Bash tool reads/writes that text without explicit encoding='utf-8'. On a Chinese-locale Windows host, CPython falls back to locale.getpreferredencoding() = cp936, often
paired with errors='surrogateescape' on stdio. The decode then splits as:

| UTF-8 byte | cp936 decode result |
|---|---|
| E2 80 | valid GBK double-byte → 鈥 (U+9225) |
| 94 | no GBK continuation → surrogateescape → U+DC94 (a lone low surrogate, PEP 383 slot) |

That string then flows back into the conversation as a role: user message, the JSONL is written with surrogatepass, and every subsequent request to the API 400s exactly as described upstream.

## Forensic evidence from a poisoned session

~/.claude/projects/<...>/<sessionId>.jsonl, line 641, role: user, position 692:

``
... # extract package 鈥\udc94 between 'anthropics_cowork_' ...
``

Raw bytes on disk:
``
e9 88 a5 # 鈥 (U+9225, valid 3-byte UTF-8)
ed b2 94 # WTF-8 of lone low surrogate U+DC94
``

That ed b2 94 triplet is exactly what str.encode('utf-8', 'surrogatepass') emits for \udc94. So Claude Code is persisting the corruption to disk with surrogatepass, then serializing it into the next
request body, where Anthropic's strict JSON parser correctly rejects it with invalid high surrogate in string.

## 5-second repro on any zh-CN / ja-JP / ko-KR Windows

``python
import subprocess
r = subprocess.run(['python', '-c', 'print("\u2014")'],
capture_output=True, text=True)
print(repr(r.stdout))
# zh-CN Windows: '鈥\udc94\n' ← lone surrogate, will poison any JSONL it lands in
# PYTHONUTF8=1 / Linux: '—\n'
``

Any code path that round-trips a child process's stdout back into the conversation (Bash tool, sub-agent transports, hook outputs, file-based agent comms) will then permanently brick the session.

## Why this expands the bug class

Prior reports in this thread (#16294, #21095, #45982, the macOS desktop Code-tab variant) blame the content of a tool's output:

• mutmut emits ANSI escapes
• Playwright snapshot contains a lone surrogate
• File read returns binary bytes
• Desktop app truncates a prefix mid-surrogate-pair

My case is categorically different: the tool's stdout was 100% valid UTF-8. The lone surrogate was manufactured by Claude Code's own subprocess decoder on a non-en-US Windows. This means:

1. The "tool emitted bad Unicode" framing is incomplete. The Windows non-UTF-8 locale codepath silently corrupts correct tool output before Claude Code ever sees it.
2. The set of affected users is much larger than "people using mutmut/Playwright/binary tools" — it's "any user on a zh-CN/ja-JP/ko-KR/etc. Windows host whose tools emit any non-ASCII character."
3. @konard's proposed sanitization is necessary as the last-line defense, but it's not sufficient — it would scrub the \udc94 after the fact,

but the conversation would still contain garbage like 鈥 where the user/tool actually meant —. The fix has to happen further upstream.

## Two-layer fix request

Layer 1 — prevent the corruption (Windows-specific, one-line):
For every subprocess spawned by the Bash tool, sub-agent transport, and hook runner, inject:

``
env.PYTHONUTF8 = "1"
env.PYTHONIOENCODING = "utf-8"
``

and pass encoding: 'utf-8' to all child_process / subprocess invocations. This deterministically eliminates the cp936-fallback origin mode without affecting Linux/macOS.

Layer 2 — defense in depth (cross-platform, the regex everyone has been asking for since June 2025):

``js
const sanitizeSurrogates = (s) =>
typeof s === 'string'
? s.replace(
/[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g,
'\uFFFD',
)
: s;
``

Applied at the tool-result boundary, before the entry is appended to the session JSONL. Also: stop writing JSONL with surrogatepass — refuse the write, or replace with U+FFFD, and surface a warning. Persisting lone
surrogates is what turns a one-shot tool failure into a permanently bricked session.

Layer 3 — recovery UX:
When the API returns invalid_request_error with "invalid (high|low) surrogate in string", auto-locate the offending message in the current JSONL, scrub, retry. Today the only recovery is manual hex surgery, which is
unreasonable on a paid product.

## User-side workaround for anyone bricked right now

``bash
python -c "
import re, shutil
p = r'<full-path-to-session>.jsonl'
shutil.copyfile(p, p + '.bak')
s = open(p, 'rb').read().decode('utf-8', 'surrogatepass')
s = re.sub(r'[\ud800-\udfff]', '', s)
open(p, 'wb').write(s.encode('utf-8'))
"
``

Then claude --resume <sessionId> works again. (Or, if you don't care about history, just start a new session.)

---

Verified against the latest docs; this still appears unresolved.

Hit this on v2.0.76 (Windows). Worth adding: for me it wasn't Bash output — the bad surrogate came from an AskUserQuestion
preview block. So sanitizing just the Bash tool result won't catch it; it needs to happen wherever the request body gets
serialized.

The bigger pain is that it doesn't just kill one turn. The bad line gets written into the session .jsonl, so every request
after that re-sends it and 400s. /compact, model switch, restart — all keep failing with the same error. The session is
basically dead.

If anyone's stuck with a bricked session: scan the transcript for the unpaired surrogate and fix it in place. Don't delete
the line — that breaks the parentUuid chain and resume loses all history. toWellFormed() handles the replacement:

const clean = s => s.toWellFormed(); // Node 20+, swaps unpaired surrogates for U+FFFD

Fixing the line in place instead of removing it let claude --resume reload the whole conversation.

I've been hitting this error a lot recently, made a hook to sanitize output: https://github.com/InfiniteLife/claude-code-surrogate-guard