[BUG] Streaming partial-JSON parser silently produces empty MCP tool arguments (accumulator shear)

Environment

• Claude Code version: 2.1.173
• Platform: Linux (x86_64, Ubuntu 24.04)
• MCP server: Custom Python FastMCP server (pydantic-validated tools)
• MCP transport: Streamable HTTP (uvicorn/starlette)

Summary

Claude Code's streaming input_json_delta accumulation pipeline silently produces {} for MCP tool arguments when the accumulated JSON buffer ends inside a string value. This is caused by a bug in the partial JSON parser (ng$ / VH1/bFH/vH1/kH1 pipeline in the bundled Anthropic SDK), not a transport or serialization issue. Once it happens, the corrupted empty args persist for all subsequent calls to that specific tool for the remainder of the session due to per-tool parameter caching in the harness.

This is the root cause behind at least #3966 and #3296.

Root Cause: Accumulator Shear in VH1 String Tokenizer

The streaming input_json_delta handler accumulates partial_json deltas into a __json_buf, then runs it through a 4-stage partial JSON parser:

ng$ = (H) => JSON.parse(kH1(vH1(bFH(VH1(H)))))

1. VH1 (tokenizer): char-by-char, produces typed tokens
2. bFH (tail trimmer): strips incomplete trailing tokens
3. vH1 (bracket closer): appends missing }/]
4. kH1 (serializer): reassembles tokens to JSON string

The bug is in VH1's string handler. When the accumulated buffer ends inside a JSON string value (the closing " hasn't arrived from the next delta), the tokenizer sets an Y=true flag and silently drops the entire string token:

// VH1 string handler (offset ~228511118 in v2.1.173 bundle)
if (K === '"') {
  let z = "", Y = false;
  K = H[++$];
  while (K !== '"') {
    if ($ === H.length) { Y = true; break; }  // buffer ends mid-string
    // ...
  }
  if (K = H[++$], !Y)
    q.push({type: "string", value: z});  // DROPPED when Y=true
  continue;
}

The cascade: Once a value string is dropped, the tail trimmer (bFH) removes the now-orphaned key token and its separator. For large payloads spanning many deltas, multiple drops cascade until the entire object body is trimmed to {}.

The silent {} path: The {} result flows through JSON.parse("{}"), becomes the content block's input field (already an object, not a string), bypasses the typeof _.input === "string" normalization branch entirely, and reaches Client.callTool({name, arguments: {}}).

Secondary Issue: Silent f = A ?? {} Fallback

When _.input IS a string that fails JSON.parse (a less common path), the normalization code at offset ~239367857 fires a telemetry event (tengu_tool_input_json_parse_fail) but then silently falls back:

if (typeof _.input === "string") {
  let A = A4(_.input, false);  // JSON.parse wrapper
  if (A === null && _.input.length > 0)
    c("tengu_tool_input_json_parse_fail", {...});
  f = A ?? {};  // SILENT FALLBACK TO {}
}

Third Issue: Per-Tool Parameter Cache Makes It Persistent

Once a tool call produces corrupted/empty input (via either path above), the Claude Code harness caches the content block's input field as the tool's parameter template. All subsequent calls to that same tool in the session reuse the cached empty args. Other tools on the same MCP server are unaffected (the cache is per-tool, not per-server).

This explains the behavior reported in #3296 where "once it gets into this state, it does not evaluate the errors returned from the MCP" and "repeats the tool call over and over with no changes to args." Session restart is the only recovery because it clears the cache.

Fourth Issue: Top-Level Backslash Handler

VH1 also has a top-level backslash handler that eats two characters:

if (K === "\") { $++; continue; }

If a streaming boundary produces a spurious top-level \ (from a truncated escape sequence), the next character is consumed without emitting a token. If that character is {, ", or ,, structure is silently lost.

Why This Disproportionately Affects Certain Tools

The partial JSON parser runs on every input_json_delta event on the accumulated buffer. Small payloads have short string values that fit in a single delta. Tools with large string arguments (our append_entry carries multi-KB markdown bodies, but any tool with substantial text input is affected) span many deltas, making it near-certain that at least one parser invocation sees an unterminated string.

Two Triggers

1. Escape interrupt during in-flight call (deterministic): user pressing Escape while a tool call is streaming leaves the content block's input in a partially accumulated state.
2. Streaming delta boundary shear (intermittent): the model's token chunking splits a JSON string value across delta boundaries, and the parser runs while the buffer contains an unterminated string.

Both leave the content block's input corrupted; the per-tool cache then makes it persistent.

Evidence

Server-side ASGI instrumentation at the transport boundary logged the events:

• body_bytes=180 (exact size of {"jsonrpc":"2.0","method":"tools/call","params":{"name":"append_entry","arguments":{}}})
• disconnect=False (no transport truncation)
• parse_err=None (valid JSON, just empty arguments)

The client sent well-formed JSON-RPC with arguments: {}. The server received it intact. The fault is entirely client-side.

Reproduction

1. Set up an MCP server with a tool that accepts a large string argument (>4KB)
2. Call the tool repeatedly in a session
3. Eventually the streaming chunking will split the string across a delta boundary, the parser drops the incomplete token, and {} reaches the server
4. All subsequent calls to that tool in the session send {} (per-tool cache)

Escape-interrupt trigger is more reliable: start a tool call with a large argument, press Escape during streaming, then call the same tool again.

Recommended Fix

1. VH1 string tokenizer: preserve incomplete (unterminated) string tokens instead of dropping them; emit a provisional token type that bFH can trim cleanly without cascading into the key/delimiter context.
2. VH1 top-level backslash handler: do not advance past the character following the backslash ($++; continue eats two chars); skip only the backslash itself so structurally significant characters aren't silently consumed.
3. Input normalization: replace f = A ?? {} with error propagation or retry. The telemetry event fires but the empty object should not reach callTool.
4. Harness parameter cache: do not cache tool parameter state derived from an aborted or errored content block.

Related Issues

• #66247 - 8x regression in tool-call parse failures since v2.1.165 (same underlying bug, measured from telemetry side); open
• #3966 - describes the same symptom (empty {} args reaching MCP servers); closed without root cause
• #4188 - Claude Desktop MCP tools failing with "Required" parameter error
• #2089 - Claude Code no longer passing parameters to MCP
• #3296 - describes the persistent-failure-then-hammering behavior (per-tool cache); closed without root cause