[BUG] Embedded CLAUDE.md intermittently reaches the API with one multibyte character replaced by three U+FFFD, changing the prompt prefix mid-session and defeating prompt caching (2.1.258, Windows)

Summary

In headless sessions (claude -p) on Windows, the text of the user's global CLAUDE.md that Claude Code embeds in the first user message (messages[0].content[0].text, inside the <system-reminder> "Codebase and user instructions are shown below") intermittently arrives at the API with one multibyte character replaced by three U+FFFD replacement characters. The file on disk is valid UTF-8 and never changes. In the affected requests the prompt prefix therefore differs from the other requests of the same session, so every such request is a full prompt-cache miss for the whole conversation.

Environment

• Claude Code 2.1.258 (native Windows build), Windows 11 22H2 (10.0.22621), PowerShell 7 host
• Global instructions file: %USERPROFILE%\.claude\CLAUDE.md, 19,597 bytes, UTF-8 without BOM, Korean prose; bytes.decode('utf-8') in strict mode succeeds
• Session form: claude -p child sessions with --allowedTools (used as workers), API traffic observed through a local proxy set with ANTHROPIC_BASE_URL that records system, messages[0] and the tool names of every outgoing request

What I observed

Over one 20-turn session, the outgoing messages[0].content[0].text came in two variants. They differ in exactly one place:

• clean variant (14 of 17 requests): '바람이 불어 창문이 흔들리는 탓에 …'
• corrupted variant (3 of 17 requests): '바람이 ���어 창문이 흔들리는 탓에 …'

difflib opcodes between the two texts: [('replace', 3615, 3616, 3615, 3618)], i.e. the single character 불 (UTF-8 EB B6 88) became three replacement characters. Nothing else in the block, the system array, or the tool list differs.

Facts about the position:

• character index 3615 of the 19,136-character block; UTF-8 byte offset 7,982 inside the block; byte offset 7,665 inside the file
• neither offset is a 4 KiB / 8 KiB / 16 KiB boundary
• the position is identical in every corrupted request across four separate sessions (two probe runs plus two long worker runs); only whether a given request is corrupted varies
• the corrupted variant appeared both as the very first request of a session and in later requests (for example requests 2, 11 and 32 of one session; 15, 24 and 26 of another), with the clean variant in between, and the file was not modified during any session

Per-request view from the proxy log of one session (sid = first 8 hex chars of sha256(system + messages[0]), in = total input tokens, cached = cached input tokens reported by the API):

msgs  sid       in     cached
  5   600ce64a 58662      0    <- first big request
  7   600ce64a 63738  58659
  9   600ce64a 63807  38874
 11   600ce64a 68896  63735
 13   600ce64a 68964  68893
 15   8fdae59a 74059      0    <- corrupted variant: different prefix, full miss
 17   600ce64a 74127  63804
 19   600ce64a 79170  68961
 21   600ce64a 79239  74124
 24   8fdae59a 84380  74056    <- corrupted variant again
 26   8fdae59a 84448      0    <- full miss
 28   600ce64a 89476  79167

Impact

Each corrupted request changes the prefix of the whole conversation, so the request misses the prompt cache for the entire context (60-100k tokens here). With the corruption on about a fifth of the requests, the sessions read 75-76% of their input from cache; with the proxy substituting the clean text for the corrupted one (a workaround, see below), the same task reads 91.6% from cache and the uncached token volume drops from about 300k to 93k. Besides cost, a request that carries � in the instructions is also a small fidelity loss.

Steps to reproduce

1. Put a UTF-8 CLAUDE.md with a few KB of Korean (or other multibyte) prose in %USERPROFILE%\.claude\ on Windows.
2. Point ANTHROPIC_BASE_URL at a local proxy that dumps messages[0] of every request (any pass-through proxy works).
3. Run claude -p "<a task that takes 15-20 tool calls>" --allowedTools Read,Grep,Bash.
4. Diff messages[0].content[0].text across the requests: in my runs 3 of 17 requests carried the corrupted character at the same position; the rest were byte-identical to the file.

I could not find the pattern that decides which request is corrupted; the file, its mtime and the session were unchanged between a clean and a corrupted request seconds apart.

Related

• #40396 (closed) reported U+FFFD for Korean characters in responses on macOS; this report is about the request side and the embedded instructions file, on Windows.
• #88836 reports U+FFFD in AskUserQuestion since 2.1.235; possibly the same decoding path.

Workaround

A proxy that remembers each clean first message it has seen and, when a request differs from a remembered text only by runs of U+FFFD, sends the remembered text instead. That keeps the prefix stable but should not be needed.

🤖 Generated with Claude Code