[BUG] Lone surrogate in Bash output produces 'unexpected end of hex escape' 400 — distinct from \u surrogate errors

Describe the bug

When a Bash tool result is a large nested JSON (e.g. from gh pr view --json), Claude Code triggers its OUTPUT_TOO_LARGE persistence path. The _generatePreview function truncates the output to a ~2KB preview. When that byte-level truncation lands inside a multi-byte UTF-8 emoji, the partial bytes are converted to a lone high surrogate (U+D83D). This surrogate is then stored in the session JSONL inside the <persisted-output> wrapper. On the next API call, the surrogate is serialized into the request body as a non-standard \x hex escape rather than a standard \u unicode escape, causing the API to reject the request with a 400 "unexpected end of hex escape" error. Once this occurs, the poisoned content is locked in the conversation context and all subsequent messages also fail.

This is a different failure mode from the "no low surrogate" / "invalid high surrogate" errors tracked in #16294 and #61301. Those errors complain about unicode escapes (\u); this one complains about hex escapes (\x).

Root cause analysis

The complete chain, reconstructed from session forensic data:

1. Bash tool outputs ~58KB of nested JSON (13 top-level fields)
2. Claude Code triggers OUTPUT_TOO_LARGE persistence
3. _generatePreview(object, firstLevelKeys, secondLevelKeys) generates a 2KB preview
4. The 2KB byte-level truncation cuts a 4-byte emoji (F0 9F 93 A5 = U+1F4E5)
   at byte 2 of 4, leaving the partial bytes F0 9F
5. The partial bytes are converted to the high surrogate U+D83D
   (U+D83D is precisely the high surrogate of U+1F4E5's surrogate pair: D83D DCA5)
6. The preview text (2256 chars) with the surrogate at position 2231 is stored
   in the session JSONL inside a <persisted-output> wrapper
7. On the next API call, the surrogate is serialized to JSON
8. Instead of standard \uD83D, the serializer emits \x hex escapes
9. DeepSeek's Anthropic-compatible API endpoint rejects the JSON:
   "unexpected end of hex escape at line 1 column 30360"
10. The poisoned content remains in context — all subsequent messages fail

Key evidence from the session JSONL:

• Tool result length: 2256 chars (truncated from 58KB)
• Surrogate U+D83D at position 2231 (25 chars before end)
• Wrapper: <persisted-output>...</persisted-output>
• Error occurred 191ms after tool result was received (parse rejection, not inference)
• Model for error: <synthetic> (system-generated, not from LLM)
• Second attempt 4 minutes later failed instantly with identical error

Binary analysis

The Claude Code 2.1.170 binary contains these relevant internal symbols:

| Symbol | Purpose |
|--------|---------|
| OUTPUT_TOO_LARGE | Threshold constant that triggers persistence |
| _generatePreview(object, firstLevelKeys, secondLevelKeys) | Generates truncated preview for structured objects |
| nonPersistentDataStore | Non-persistent data storage path |

These confirm that a dedicated preview generation code path exists for structured tool output, distinct from the general text truncation path.

Changelog partial fix

The 2.1.170 changelog includes:

Fixed API 400 no low surrogate in string errors for classifier side-queries and MCP server descriptions containing emoji near a truncation boundary

This fix covered classifier and MCP tool description paths, but did not cover the Bash tool output persistence preview path (_generatePreview). The underlying mechanism is identical: byte-level truncation near a multi-byte UTF-8 character produces a lone surrogate that corrupts JSON serialization.

Why this is NOT a duplicate of #16294 / #61301

Those issues report errors from the JSON parser's \u unicode escape handler ("no low surrogate", "invalid high surrogate"). This issue reports an error from the parser's \x hex escape handler ("unexpected end of hex escape"). Different parser code path, different serialization bug, same trigger class (lone surrogates from truncation).

Reproducibility

This bug cannot be reliably reproduced in 2.1.170 because the OUTPUT_TOO_LARGE threshold appears to have been raised or the preview generation behavior was modified. Tool outputs up to 200KB are now displayed inline without triggering the <persisted-output> preview path. However, the _generatePreview function and OUTPUT_TOO_LARGE constant remain in the binary — if the threshold is ever lowered or the preview path is re-enabled for Bash output, the bug will resurface.

Fix recommendation

The fix applied to classifier and MCP paths (sanitizing surrogates before JSON serialization) should be extended to cover all tool output paths, including the _generatePreview persistence preview path. Specifically, any string content that passes through the <persisted-output> wrapper should have lone surrogates replaced with U+FFFD before being stored in the session JSONL.