Auto-compaction: two compounding cost bugs — stale precompute keeps ~200k tokens verbatim, and that prefix is repeatedly cache-created instead of cache-read

Summary

Two separate but compounding cost problems in the auto-compaction path. The first bloats the conversation; the second then pays to re-ingest that bloat repeatedly.

1. Stale precompute → shallow compaction. /compact consumes a background precompute summary that was baked ~47 minutes earlier. Consuming it only summarizes up to the old anchor and preserves every message since (~200k tokens) verbatim, so compaction barely reduces context. There's no staleness/tail-size guard that would rebuild the precompute or fall back to a fresh summary.
2. **Cache-sharing miss → the bloated prefix is cache-created instead of cache-read. Because the main-thread request anchors its body cache_control breakpoint on the last (volatile) message, the background precompute (and the following turn) cannot reuse the cached body — they re-create the full ~196k at the cache write rate ($10/MTok 1h) instead of the read** rate ($0.50/MTok), ~20× more, and it recurs on every precompute.

Environment

• Claude Code version: 2.1.186 (claude-cli/2.1.186, native install)
• Model: claude-opus-4-8
• Platform: macOS (darwin arm64), Node v24.3.0
• Relevant settings (settings.json → env, possibly influencing compaction trigger frequency):

"CLAUDE_CODE_DISABLE_1M_CONTEXT": "1",
"CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "80"

(The first caps the context window at 200k instead of 1M; the second sets the auto-compact threshold to 80% of the window — together they make auto-compaction/precompute fire earlier and more often.)

Issue 1 — Stale precompute kept ~200k tokens verbatim

Precompute lifecycle from the debug log:

20:13:24 precomputed compact: started (main, 211 msgs, ~147k tok, attempt 2, trigger api_response)
20:15:07 precomputed compact: ready   (103s)
21:02:46 precomputed compact: consumed (main, ready)   <- consumed 47 min after it was built

• The /compact ran at 21:02:45; the summary it consumed is timestamped 20:15:07 — a 47-minute-old summary.
• ~250 messages (180 assistant / 70 user) were created in that 47-minute window. The consume path preserves all messages-since-anchor verbatim (messagesToPreserve = [...precompute.messagesToPreserve, ...messagesSinceAnchor]), so none of them are summarized.
• Result: the post-compact conversation stays huge — observed at the next precompute as 452 messages / ~238k tokens. A fresh summarization at 21:02 would have collapsed those ~250 messages into a short summary instead.

Why it's a bug: a precompute is consumed regardless of how stale it is or how large the verbatim tail has grown. When the tail is ~200k tokens, the "compaction" frees almost nothing.

Issue 2 — The bloated prefix is cache-created, not read

Two back-to-back requests, 9s apart, on that ~238k-token conversation (162 messages):

REQ1 — main-thread user turn (source=repl_main_thread):

"usage":{"input_tokens":1494,"cache_creation_input_tokens":196093,"cache_read_input_tokens":39766,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":196093},"output_tokens":312,"service_tier":"standard","inference_geo":"not_available"}

REQ2 — background precompute (source=compact, forked reactive-compact):

"usage":{"input_tokens":3737,"cache_creation_input_tokens":195784,"cache_read_input_tokens":39766,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":195784},"output_tokens":6549,"service_tier":"standard","inference_geo":"not_available"}

Debug log for REQ2:

21:16:55 precomputed compact: started (main, 452 msgs, ~237,765 tok, attempt 3, trigger api_response)
21:16:56 [API REQUEST] /v1/messages source=compact
21:18:11 reactive-compact finished: cacheRead=39766 cacheCreate=195784
21:18:11 precomputed compact: ready (main, 75333ms)

cache_read_input_tokens is pinned at 39,766 (system prompt + tool defs only) in both; the entire ~196k message body is cache-created both times. Structural diff of the two request bodies:

• Messages [0..159] are content-identical.
• They diverge only at messages[160] (the final user turn): the precompute appends a CRITICAL: Respond with TEXT ONLY... + <analysis>/<summary> instruction.
• cache_control body breakpoint: REQ1 on messages[160] (the volatile last message); REQ2 on messages[159].
• The main thread only cached up to messages[160] (which the precompute then modifies), so the precompute has no usable checkpoint at messages[159] and re-creates from the system head.

Why it's a bug: the precompute is intended to cache-read the conversation (tengu_compact_cache_prefix / tengu_compact_cache_sharing). The breakpoint-on-the-volatile-message placement defeats that, so the share misses and the body is re-created. The tengu_compact_cache_sharing_fallback event is presumably firing; its reason would quantify fleet-wide frequency.

Impact

• Issue 1: every post-compaction turn carries ~200k unnecessary tokens (≈ +$0.10/turn in cache reads at $0.50/MTok, plus larger periodic re-creates), and the next precompute must summarize a needlessly large ~238k context.
• Issue 2: each background precompute re-creates the full body at write rates — ~196k × ($10 − $0.50)/MTok ≈ ~$1.9 of avoidable spend per precompute. This session ran the precompute 3 times (18:22, 20:13, 21:16) → ~$5+ avoidable, and it scales with conversation size (so the longest sessions pay most). The single pair above cost ≈ $4.16, ~$3.9 of it avoidable cache-creation.

Suggested fixes

1. Issue 1 — add a staleness / tail-size guard. When the messages-since-precompute exceed a token threshold (or the precompute is older than N minutes / M messages), rebuild the precompute or fall back to a fresh live summarization, so /compact actually reduces context instead of preserving ~200k verbatim.
2. Issue 2 — align cache breakpoints. Either have the main-loop request place a breakpoint at the last committed message boundary (before the in-flight turn) so forks/precomputes can read up to it, or have the precompute reuse the main thread's existing breakpoint and append its instruction after that boundary — so only the small instruction delta is cache-created.

Repro / detection

• Long Opus session with auto-compact enabled, until precomputed compact: started fires.
• Issue 1: compare the consumed summary's timestamp to the /compact time, and check how many messages were preserved verbatim.
• Issue 2: on the source=compact request, cache_creation_input_tokens ≈ full context while cache_read_input_tokens ≈ only the system prompt indicates the miss. Check tengu_compact_cache_sharing_fallback for frequency/reason.