[BUG] High Token Burn Due to Redundant Context Resubmission & Compaction Loops

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

The current query + compaction pipeline is causing severe token inefficiency (50K–300K+ tokens per event) due to repeated full-context resubmissions, ineffective retry handling, and cascading compaction behavior. These issues compound under error conditions, leading to exponential token waste. See below:

1. Query Loop Retries Resend Full Context

Impact: 50K–300K tokens per retry

Problem:

• The main while (true) query loop resends the entire message history, system prompt, and tool schemas on every retry.
• Retries are triggered by:
• prompt-too-long errors
• max-output-token limits
• streaming/media failures
• compaction failures
• No deduplication or reuse of unchanged context.

Fix:

• Track whether context has changed between iterations.
• Introduce a contextHash:
• Compute hash of serialized messages + system prompt + tools.
• Compare before each API call.
• If the hash is unchanged:
• Skip the API call or apply backoff instead of resending.
• Add an early-exit condition:
• If retry is triggered but context has not grown/changed, do not reserialize or resend.

---

2. Autocompact Cascade Uses Full Context

Impact: 100–200K tokens per compaction (up to 3× per turn)

Problem:

• Autocompact triggers at ~187K tokens and submits the entire bloated context for summarization.
• The loop can trigger repeatedly (up to MAX_CONSECUTIVE_AUTOCOMPACT_FAILURES).
• No validation that compaction meaningfully reduces token count.
• Can immediately re-enter compaction after finishing.

Fix:

• Trigger compaction earlier:
• At ~70–75% of context window instead of near max capacity.
• Enforce a minimum reduction threshold:
• Require ≥30% token reduction; otherwise treat as failure and abort cascade.
• After compaction:
• Recalculate token count before re-entering query loop.
• Prevent immediate re-trigger unless new tokens exceed threshold again.
• Add guard against cascading:
• Limit consecutive compactions more aggressively or introduce cooldown between attempts.

---

3. Streaming Fallback Duplicates Full Requests

Files:
Impact: 50–200K tokens per fallback

Problem:

• On streaming failure (e.g., 529 errors), system falls back to non-streaming mode.
• The fallback resends the entire message array from scratch.
• Partial streamed output is discarded.

Fix:

• Implement resume-from-checkpoint:
• Capture partial streamed tokens before failure.
• Append them to context for retry instead of restarting.
• Retry streaming before fallback:
• Add exponential backoff for transient failures.
• Add deduplication:
• If the same contextHash is about to be resent within the same turn, delay or skip retry.
• Only fallback to non-streaming after multiple streaming retries fail.

---

4. Tool List Changes Invalidate Prompt Cache

Impact: 50–200K tokens per cache bust

Problem:

• Deferred tool updates prepend a new message mid-turn.
• This invalidates prompt caching, forcing full context reprocessing.
• Even minor differences (e.g., ordering) trigger cache busts.

Fix:

• Batch tool list updates:
• Apply only at turn boundaries, not mid-turn.
• Normalize tool definitions:
• Sort tools alphabetically before serialization.
• Ensure stable formatting to prevent unnecessary diffs.
• Add change detection:
• Compare new tool list against previous version.
• Skip reinsertion if there is no meaningful change.
• Avoid prepending new messages unless strictly required.

---

Combined Effect

These issues amplify each other:

• Retry loops + streaming fallback → duplicate full requests
• Compaction loops → repeated large summarization calls
• Tool updates → cache invalidation → no reuse of prior computation

Result: Massive, avoidable token burn and degraded performance under load or failure scenarios.

What Should Happen?

Implementing these fixes should:

• Reduce token usage per retry/failure by 50–90%
• Eliminate redundant full-context resubmissions
• Prevent cascading compaction loops
• Improve latency and overall system stability
• Significantly reduce API costs

Error Messages/Logs

Steps to Reproduce

1. Run 'claude'
2. Prompt and observe abnormally high token consumption

Claude Model

None

Is this a regression?

No, this never worked

Last Working Version

_No response_

Claude Code Version

2.1.9

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

Terminal.app (macOS)

Additional Information

_No response_