Intermittent 400 'surrogates not allowed' when context contains emoji (astral chars)

Title

Intermittent 400 str is not valid UTF-8: surrogates not allowed when context contains emoji (astral/supplementary-plane characters)

Environment

• Claude Code VSCode extension (anthropic.claude-code), versions 2.1.197 and 2.1.198 — bug persisted across the update
• Windows 11, locale es-AR (system non-Unicode/ANSI codepage = Windows-1252), terminal codepage UTF-8 (chcp 65001)
• Not reproducible only via image paste — confirmed the same error also occurs on text-only requests with no image attached at all

Symptom

API Error: 400 The request body is not valid JSON: str is not valid UTF-8: surrogates not allowed: line 1 column 1 (char 0), intermittent, across many different unrelated tasks/sessions throughout the day. Retried automatically (attempt 1/11 in logs) and next attempt usually succeeds.

Evidence gathered

Checked %APPDATA%\Code\logs\<session>\window1\exthost\Anthropic.claude-code\Claude VSCode.log. The 400 recurs across sessions with unrelated titles/content (e.g. a "list memory directory" task), ruling out clipboard/image-paste as the trigger.

The open project (JiraToolkit) has two working-tree files, js/dashboard.js and js/taskmodal.js, containing ~22 literal emoji characters outside the Basic Multilingual Plane (📭 U+1F4ED, 🌴 U+1F334, 🎯 U+1F3AF, 👥 U+1F465, 🗑 U+1F5D1, etc.). These are represented in JS/UTF-16 as surrogate pairs (two 16-bit code units). Any operation that slices such a string by raw code-unit count without respecting surrogate-pair boundaries (context-window truncation, a bounded preview of a tool result/diff, message compaction, etc.) can land in the middle of a pair and leave one unpaired surrogate in the string. When that string is later JSON-serialized into the /v1/messages request body, the resulting bytes are not valid UTF-8/WTF-8, and the API correctly rejects the whole body — explaining why the error always points at char 0 (a whole-body decode failure) rather than a specific field, and why it's intermittent rather than deterministic per-request.

Workaround applied locally

Replaced the literal emoji with JS \u{XXXXX} escapes in source (renders identically at runtime, but the raw file text no longer contains an actual surrogate pair for Claude Code to potentially split). This is a workaround for our project, not a fix for the extension.

Suggested fix

Wherever Claude Code truncates/windows text before building the API request body (tool-result truncation, diff previews, context compaction, message-history windowing), ensure truncation is surrogate-pair-aware (e.g. use Array.from(str) / grapheme-aware slicing, or check str.codePointAt boundaries) instead of slicing raw UTF-16 code units. Alternatively/additionally, sanitize outgoing request bodies by replacing any lone surrogate with U+FFFD before JSON-encoding, as defense in depth.

Request IDs from affected calls (for server-side log lookup)

• req_011Ccc6YMvdNYGm8JjpcmgiY
• req_011Ccc6ouKBJaBXtehNPSdf6
• req_011Ccc6pstdYcPaoz5tovsKw
• req_011Ccc6xKXBbb2EJ7rwh4ivV
• req_011Ccc7N29q5Yy2nV5iAnkWA
• req_011Ccc7bt72xn5YaACbwLjAH