Prompt cache: first-message context envelope (~20k tok) never reuses across sessions — MCP connect race + per-session assembly defeat caching

Summary

Cross-session prompt-cache reuse fails for the first-message context envelope (~19–20k tokens on a configured setup), even when its content is byte-deterministic. Two distinct causes observed on 2.1.170 (macOS):

1. MCP connect race makes the deferred-tools snapshot nondeterministic. The deferred_tools_delta block injected at session start reflects whichever MCP servers happen to have connected by snapshot time. Two consecutive identical claude -p "<same prompt>" runs in the same cwd produced deltas of 8,106 vs 873 chars — one run listed 78 remote-connector tools, the next listed none. A single divergent block early in the first message forces the entire downstream envelope (user CLAUDE.md, skill listing, hook outputs — ~20k tokens here) to re-write to cache every session.

1. Even a fully deterministic envelope is never reused across sessions. With --strict-mcp-config (empty pinned MCP set) and deterministic SessionStart hook output, two consecutive identical runs both report cache_creation_input_tokens: 19303 — byte-identical size, zero reuse — while cache_read_input_tokens stays pinned at 18,607 (the system-prompt block, which DOES reuse across sessions). Something per-session in the assembled first message (internal IDs on injected attachments? breakpoint placement?) appears to defeat the cache lookup even when all user-visible content is identical.

Repro

cd /tmp/scratch   # no project CLAUDE.md
claude -p "Explain the difference between optimistic and pessimistic locking." \
  --model sonnet --output-format json | jq .usage
# run it again within the cache TTL:
claude -p "Explain the difference between optimistic and pessimistic locking." \
  --model sonnet --output-format json | jq .usage

Observed (run 2, expected near-full cache read): cache_creation_input_tokens ≈ 19–21k, cache_read_input_tokens = 18,607 (system prompt only). Comparing the two session transcripts shows the deferred_tools_delta attachment as the largest content difference; with --strict-mcp-config the content goes deterministic but reuse still doesn't happen.

For comparison, --safe-mode runs converge to full reuse by run 2 (cache_creation → 0), so cross-session caching works when the customization envelope is absent.

Impact

For a user with a typical power configuration (user CLAUDE.md + hooks + plugins + several MCP servers), every new session re-writes ~20k tokens at cache-write rates (1.25–2× input). Measured on a fixed 5-prompt sweep: warm-cache cost was 6.0× the --safe-mode baseline, almost entirely from this envelope churn. Short-session-heavy workflows (headless -p fleets, cron jobs, subagent spawns) pay it on every invocation.

Suggestions

• Snapshot deferred tools deterministically: stable ordering derived from configuration rather than connect-state at an arbitrary instant (or delay the snapshot until connects settle / make the delta block position-stable at the end of the envelope).
• Audit the first-message assembly for per-session identifiers ahead of static content, and consider ordering static-before-dynamic so the stable bulk (CLAUDE.md, skill listing) shares a cache prefix across sessions.

Happy to provide the full usage JSONs and transcript diffs.