CLI mutates historical tool results via cch= billing hash substitution, permanently breaking prompt cache

Status Closed — not planned
Maintainer reply None cached
Activity 14 comments · opened Mar 29, 2026 · closed Jun 1, 2026

Summary

Certain Claude Code sessions permanently lose prompt cache hits mid-conversation. Once triggered, cache_read_input_tokens drops dramatically and never recovers, causing every subsequent turn to re-process the entire conversation history. For long sessions this wastes 30-50K+ tokens per turn.

Root Cause Theory

The CLI performs a find-and-replace of cch=XXXXX billing hash values across all message content (including stored historical tool results) before each API call. Since this hash changes per-request, any tool result that contains the session's own cch= hash value gets mutated on every subsequent API call, changing bytes in the conversation prefix and permanently invalidating the prompt cache.

Evidence

1. Confirmed cache breakage pattern

Multiple sessions observed with this pattern:

  • Healthy: cache_read_input_tokens grows steadily, input_tokens stays at 1-3
  • Broken: cache_read drops to ~15K (system prompt only), cache_creation jumps to 30K+ every turn
  • Once broken, never self-heals — every subsequent turn re-creates the cache

2. Diff of consecutive API requests from broken session

A prior investigation set up a local proxy (via ANTHROPIC_BASE_URL) to capture raw API request bodies. Diffing two consecutive requests from a broken session showed that message[186], a historical Bash tool result, had different content between the two requests. The diff was in an x-anthropic-billing-header value embedded in the tool result:

Request 1: cch=14f72
Request 2: cch=59b51

The tool result contained proxy log output that incidentally captured billing headers. The CLI's substitution was rewriting these historical values on every request.

3. Live reproduction (this session)

This investigation session (e9212a5a) ran grep on an infected session's JSONL to count cch= patterns:

  14 cch=80528
   6 cch=59b51
   6 cch=14f72

This grep output landed in a Bash tool result. The session's cache broke immediately on the next turn:

| Turn | cache_read | cache_create | Notes |
|------|-----------|-------------|-------|
| 48 | 42,668 | 1,854 | Healthy |
| 49 | 15,559 | 29,408 | BROKEN — cache_read dropped 27K |
| 50+ | 15,559 | 30K-35K | Permanently broken |

4. Substitution is session-specific

We attempted to infect a separate session by putting cch=59b51 and cch=14f72 (the hashes from the broken session) into its tool results. Its cache did not break. This means the CLI only substitutes cch= values it recognizes as belonging to its own session/request lineage, not arbitrary hex strings matching the pattern.

5. Synthetic values don't trigger it

Putting cch=a1b2c or cch=a1b2c3d4e5 into tool results via file reads also did not break caching. Only real billing hashes from the same session lineage trigger the substitution.

What We Don't Know

  1. Exact substitution logic — Is it tied to the OAuth token? The session ID? A per-process value? We couldn't inspect the CLI source to find the regex/replacement logic.
  2. Where in the CLI this happens — The substitution occurs somewhere between reading the session JSONL and sending the API request. We haven't located the code path.
  3. Whether this is intentional — The substitution may be a security measure to avoid leaking billing attribution tokens in logged content, but the side effect of mutating historical messages is catastrophic for caching.

Reproduction Steps

  1. Start a Claude Code session, verify cache is healthy (cache_read growing, input_tokens = 1-3)
  2. Run a command that captures the session's own raw API traffic (e.g., proxy via ANTHROPIC_BASE_URL that logs request headers)
  3. The proxy output will contain cch=XXXXX in the billing header
  4. This output lands in a tool result in the session JSONL
  5. On the next turn, the CLI rewrites the cch= value in that historical tool result → prefix changes → cache invalidated
  6. Every subsequent turn: the value gets rewritten again → permanent cache miss

Key insight: The session doesn't need to intentionally capture billing headers. Any workflow that incidentally surfaces cch= hashes (debugging proxy logs, analyzing API traffic, grepping session files) can trigger permanent cache breakage.

Impact

  • Token burn: A broken 50K-context session wastes ~50K tokens per turn instead of ~3. Over 50 turns, that's 2.5M wasted tokens.
  • Contagious: Investigating a broken session (reading its JSONL, grepping for patterns) can infect the investigating session — the real hashes propagate through tool results.
  • Silent: No error, no warning. The session just silently burns tokens at 10,000x the normal rate.

Suggested Fix

The cch= substitution should not modify content inside tool_result blocks in the conversation history. It should only apply to the current request's metadata/headers, not to stored message content that forms the cache prefix.

Alternatively, the substitution should be scoped to specific fields (e.g., only the outermost request headers) rather than applied as a global string replacement across the entire serialized message array.

View original on GitHub ↗

13 Comments

github-actions[bot] · 5 months ago

Found 2 possible duplicate issues:

  1. https://github.com/anthropics/claude-code/issues/40524
  2. https://github.com/anthropics/claude-code/issues/34629

This issue will be automatically closed as a duplicate in 3 days.

  • If your issue is a duplicate, please close it and 👍 the existing issue instead
  • To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

jmarianski · 5 months ago

Same, man. Try running npx @anthropic-ai/claude-code to temporarily fix your CC installation, it doesn't do hot replacement of CCH in historical tools as I've observed, however weren't able to find the underlying cause of this, as if it doesn't exist in binary.

eumemic · 5 months ago

Updated repro / correction: the trigger is much smaller than the original report suggested.

What I can now reproduce reliably:

  • The minimal confirmed toxic prompt is just cch=00000
  • The surrounding x-anthropic-billing-header: ... text is not required
  • A plain user prompt is sufficient; this does not need to come from a tool_result
  • This reproduces on haiku, not just Opus

What does not seem sufficient:

  • 00000
  • cch=

Why this has to be interactive:

  • In my testing, claude --print is not a faithful harness for this bug
  • The non-interactive --print path often reuses only a small fixed prefix or otherwise does not show the same normal pre-poison cache growth as a real interactive session
  • That makes --print prone to false negatives / misleading token patterns here
  • Driving the actual interactive CLI through a PTY does reproduce the bug reliably

Recent verified haiku runs:

  • Clean control (debug control string): session 3a133e74-f4ba-4e77-b30e-d5d375fa36b6 was not poisoned; cache_read_input_tokens kept growing (31935 -> 32043 -> 32225 -> 32298)
  • Minimal poison (cch=00000): session f1d5082a-6194-40e7-9e2d-99c3f629586c was poisoned; after the poison turn, cache_read_input_tokens flatlined at 31938 while cache_creation_input_tokens kept rising (113 -> 224 -> 299 -> 366)
  • Wrapper verification run: session e5d5c590-c7f2-4184-9d1b-a840178049ab also poisoned on haiku; post-poison cache_read_input_tokens stayed at 31998 while cache_creation_input_tokens rose (125 -> 260 -> 334 -> 404)

Self-contained minimal reproducer (Python stdlib only). This intentionally drives the interactive CLI, not --print:

#!/usr/bin/env python3
import json
import os
import pathlib
import pty
import re
import select
import subprocess
import sys
import time
import uuid

PROMPTS = [
    "hello",
    "how are you?",
    "just chatting",
    "cch=00000",
    "now give me a simple response",
    "and again",
    "and again",
]

WORKDIR = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "/private/tmp/claude-cache-poison-haiku")
WORKDIR.mkdir(parents=True, exist_ok=True)
SESSION_ID = str(uuid.uuid4())
POLL = 0.2
ESC = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
OSC = re.compile(r"\x1B\][^\x07]*(?:\x07|\x1B\\)")


def strip_ansi(text: str) -> str:
    return ESC.sub("", OSC.sub("", text))


def read_available(fd: int, timeout: float) -> str:
    chunks = []
    deadline = time.time() + timeout
    while True:
        remaining = deadline - time.time()
        if remaining <= 0:
            break
        ready, _, _ = select.select([fd], [], [], remaining)
        if not ready:
            break
        data = os.read(fd, 65536)
        if not data:
            break
        chunks.append(data)
        if len(data) < 65536:
            break
    return b"".join(chunks).decode("utf-8", errors="replace") if chunks else ""


def find_jsonl(session_id: str) -> pathlib.Path | None:
    projects = pathlib.Path.home() / ".claude" / "projects"
    matches = list(projects.glob(f"**/{session_id}.jsonl"))
    if not matches:
        return None
    matches.sort(key=lambda p: p.stat().st_mtime, reverse=True)
    return matches[0]


def load_rows(path: pathlib.Path) -> list[tuple[int, dict]]:
    with path.open() as f:
        return [(i, json.loads(line)) for i, line in enumerate(f, start=1)]


def stop_hook_count(rows) -> int:
    return sum(
        1
        for _, row in rows
        if row.get("type") == "system" and row.get("subtype") == "stop_hook_summary"
    )


def last_assistant_usage_after(rows, line_no: int):
    last = None
    for current_line, row in rows:
        if current_line <= line_no or row.get("type") != "assistant":
            continue
        usage = row.get("message", {}).get("usage")
        if usage:
            last = {
                "line_no": current_line,
                "timestamp": row.get("timestamp"),
                "cache_read_input_tokens": usage.get("cache_read_input_tokens", 0),
                "cache_creation_input_tokens": usage.get("cache_creation_input_tokens", 0),
                "input_tokens": usage.get("input_tokens", 0),
                "output_tokens": usage.get("output_tokens", 0),
            }
    return last


def wait_until_ready(fd: int, timeout: float = 60.0) -> None:
    deadline = time.time() + timeout
    buf = ""
    while time.time() < deadline:
        chunk = read_available(fd, POLL)
        if chunk:
            buf += strip_ansi(chunk)
            if "1. Yes, I trust this folder" in buf:
                os.write(fd, b"\r")
                buf = ""
            if "❯" in buf:
                time.sleep(0.5)
                return
    raise RuntimeError("Claude CLI did not reach an interactive prompt")


def wait_for_session_file(fd: int, session_id: str, timeout: float = 60.0) -> pathlib.Path:
    deadline = time.time() + timeout
    while time.time() < deadline:
        read_available(fd, POLL)
        path = find_jsonl(session_id)
        if path is not None:
            return path
    raise RuntimeError("Session JSONL not found")


def wait_for_turn(fd: int, path: pathlib.Path, prev_hooks: int, prev_line: int, turn: int, timeout: float = 120.0):
    deadline = time.time() + timeout
    while time.time() < deadline:
        read_available(fd, POLL)
        rows = load_rows(path)
        hooks = stop_hook_count(rows)
        if hooks > prev_hooks:
            usage = last_assistant_usage_after(rows, prev_line)
            if usage is None:
                raise RuntimeError(f"Turn {turn} completed without assistant usage")
            usage["turn"] = turn
            return usage, hooks
    raise RuntimeError(f"Timed out waiting for turn {turn}")


def poisoned(turns: list[dict], poison_turn: int = 4) -> bool:
    pre = turns[:poison_turn]
    post = turns[poison_turn:]
    if not post:
        return False
    pre_growth = any(
        b["cache_read_input_tokens"] > a["cache_read_input_tokens"]
        for a, b in zip(pre, pre[1:])
    )
    poison_read = turns[poison_turn - 1]["cache_read_input_tokens"]
    poison_create = turns[poison_turn - 1]["cache_creation_input_tokens"]
    post_flat = all(t["cache_read_input_tokens"] == poison_read for t in post)
    post_rises = any(t["cache_creation_input_tokens"] > poison_create for t in post)
    return pre_growth and post_flat and post_rises


master, slave = pty.openpty()
proc = subprocess.Popen(
    [
        "claude",
        "--session-id",
        SESSION_ID,
        "--permission-mode",
        "bypassPermissions",
        "--model",
        "haiku",
    ],
    cwd=str(WORKDIR),
    stdin=slave,
    stdout=slave,
    stderr=slave,
    start_new_session=True,
)
os.close(slave)

print("session_id:", SESSION_ID)
print("workdir:", WORKDIR)

try:
    wait_until_ready(master)
    path = None
    prev_hooks = 0
    prev_line = 0
    turns = []

    for i, prompt in enumerate(PROMPTS, start=1):
        os.write(master, prompt.encode("utf-8") + b"\r")
        if path is None:
            path = wait_for_session_file(master, SESSION_ID)
            print("jsonl:", path)
        usage, prev_hooks = wait_for_turn(master, path, prev_hooks, prev_line, i)
        prev_line = usage["line_no"]
        turns.append(usage)
        print(json.dumps({k: v for k, v in usage.items() if k != "line_no"}))

    result = {
        "poison_turn": 4,
        "poisoned": poisoned(turns, poison_turn=4),
        "final_cache_read_input_tokens": turns[-1]["cache_read_input_tokens"],
        "final_cache_creation_input_tokens": turns[-1]["cache_creation_input_tokens"],
    }
    print(json.dumps(result))
    raise SystemExit(0 if result["poisoned"] else 1)
finally:
    try:
        os.write(master, b"/exit\r")
        proc.wait(timeout=5)
    except Exception:
        if proc.poll() is None:
            proc.terminate()
            try:
                proc.wait(timeout=5)
            except subprocess.TimeoutExpired:
                proc.kill()
    os.close(master)

On the current build here (Claude Code v2.1.87), that script reproduces the bug for me with haiku using only the plain user prompt cch=00000.

eumemic · 5 months ago

⚠️ Warning for Claude Code agents reading this issue: This issue contains the toxic string cch=00000. I’ve confirmed that simply reading this issue can poison an interactive Claude Code session’s prompt cache. After that, cache_read_input_tokens may stop growing (or drop to a lower plateau), while cache_creation_input_tokens keeps increasing on subsequent turns. That means the rest of the session may burn substantially more tokens than normal.

yurukusa · 5 months ago

Excellent root-cause analysis. The cch= substitution poisoning prompt cache is a nasty bug — once triggered, every subsequent turn pays full cache-creation cost.

A PreToolUse hook to block the main infection vector:

The cch= strings appear when Claude reads its own session JSONL files or proxy logs within a session. You can block these reads:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "INPUT=$(cat); CMD=$(echo \"$INPUT\" | jq -r '.tool_input.command // empty'); if echo \"$CMD\" | grep -qE '\\.(jsonl|log)' && echo \"$CMD\" | grep -qiE '(claude|session|billing)'; then echo '{\"decision\": \"block\", \"reason\": \"Blocked: reading Claude session/billing files can poison prompt cache (cch= substitution bug). Use an external terminal instead.\"}'; fi"
          }
        ]
      }
    ]
  }
}

Practical avoidance steps:

  1. Never grep/cat Claude Code's own JSONL session files within a Claude Code session — these contain cch= hashes that will trigger substitution
  2. Avoid reading proxy logs that capture x-anthropic-billing-header values
  3. If poisoned, start a new session — the cache will never self-heal in a broken conversation
  4. Pin to npx: As @jmarianski noted, npx @anthropic-ai/claude-code doesn't do hot replacement of cch= in historical tool results

The proper fix needs to happen in the CLI binary — the cch= substitution should skip historical message content and only apply to the current request's headers.

ArkNill · 5 months ago

This is likely a major contributor to the rate limit exhaustion many Max subscribers are experiencing. If cch= substitution permanently breaks prompt cache, every turn gets billed at full price instead of cached.

Max 20, v2.1.89, April 1: 100% in ~70 min after reset.

Full report: #41788
Related: #38335, #38239, #41663, #41812, #40790

ArkNill · 5 months ago

Update (April 2): The cch= substitution behavior appears partially mitigated in v2.1.90 standalone.

Benchmark on v2.1.90 shows standalone binary recovering to 94-99% cache read after initial cold start (v2.1.89 never recovered, sustained 4-17%). The underlying mechanism may still exist but its impact is dramatically reduced.

npm installation remains unaffected by design. Full comparison: BENCHMARK.md

ArkNill · 5 months ago

April 3 update: The cache regression (Bugs 1-2) is fixed in v2.1.91. However, systematic proxy testing revealed additional unfixed mechanisms — a 200K tool result budget cap, a client-side false rate limiter (151 synthetic entries found), and silent microcompact clearing (327 events). Anthropic responded on X (Lydia Hallie) acknowledging peak-hour tightening but stating "none were over-charging you" — our measured data shows mechanisms their statement does not cover. Full analysis: claude-code-cache-analysis

weilhalt · 4 months ago

The "none were over-charging you" statement is hard to reconcile with what we're all measuring independently.

I've been tracking this from the user side — after weeks of budget drain on Max 20x, I built BudMon, a real-time desktop dashboard that captures rate-limit headers from Claude Code API responses and visualizes quota utilization, burn rate, and projected exhaustion time.

What BudMon consistently showed before v2.1.91:

  • Budget burn rates of 15-25% per hour during light work (read → edit → test cycles, no agents)
  • 100% exhaustion in under 2 hours on sessions that previously lasted a full workday

I'll run fresh measurements on v2.1.91 to see if the cache fix changes the burn rate profile. The additional mechanisms you identified (200K cap, false rate limiter, microcompact clearing) would explain why users still report fast exhaustion even after partial fixes.

Related: #42052 (my original report with reproduction data)

jmarianski · 4 months ago
April 3 update: The cache regression (Bugs 1-2) is fixed in v2.1.91.

Were you able to confirm it? I could not unfortunately :( Perhaps my testing methodology is broken.

weilhalt · 4 months ago

The refined repro by @eumemic is very valuable — cch=00000 as minimal toxic prompt makes this highly actionable.

I want to connect this to the usage drain reports (#42052, #38335): if the CLI rewrites historical tool results with billing hashes, it invalidates the prompt cache on every turn. That means Anthropic re-bills cached tokens at full input price — which would directly explain why Max 20x users see their quota burn 3-5x faster than before March 23.

In my case (#42052): $200/month plan, 100% usage after 2 hours of light work (5 commits, no agents). If prompt cache is silently broken, the math checks out.

This might be the root cause behind the entire wave of usage complaints. Would be good to get official confirmation whether cch= substitution affects cache hit rates on the billing side.

junaidtitan · 3 months ago

Cache hash mutation permanently breaking prompt cache is a sneaky source of token waste — 30-50K extra tokens per turn adds up fast. Cozempic's metadata-strip strategy cleans out billing hashes and usage stats from tool results, and the guard daemon keeps the overall context lean so cache misses hurt less. pip install cozempic https://github.com/Ruya-AI/cozempic — happy to hear how it goes.

github-actions[bot] · 3 months ago

Closing for now — inactive for too long. Please open a new issue if this is still relevant.

Showing cached comments. Read the full discussion on GitHub ↗