Session-level Context-Augmented Generation (CAG) as alternative to lossy compaction

Problem Statement

Current compaction summarizes the entire conversation history into a compressed summary when context approaches capacity (~83.5% of 200K). This is fundamentally lossy — fine-grained details like variable names, schema column decisions, specific error messages, architectural rationale, and implementation nuances get dropped. For complex multi-file tasks (e.g., implementing a feature across a monorepo with mobile and web clients), post-compaction Claude frequently:

• Loses track of schema decisions made earlier in the session
• Re-reads files it already processed
• Contradicts its own prior implementation choices
• Forgets edge cases and constraints discussed in detail

The Memory Tool (API-level) and CLAUDE.md provide partial workarounds, but they require Claude to proactively decide what's worth saving before compaction fires. This is the wrong inversion — the system should preserve everything and retrieve selectively, not discard everything and hope the summary captured what matters.

Proposed Solution

Implement a session-level Context-Augmented Generation (CAG) system as an alternative (or complement) to summarization-based compaction:

Core concept: Instead of summarizing old context into a lossy blob, persist full conversation turns to a local session store (structured files or lightweight embedded index). When context needs to be freed, evict older turns from the active window but retain them in the store. When Claude needs information from earlier in the session, retrieve relevant turns on demand.

How it could work

1. Sliding window + retrieval hybrid — Keep the most recent N turns verbatim in the context window (sliding window). Older turns get moved to a local session store indexed by content.

1. On-demand retrieval — When Claude's response or tool use references something not in the active window, trigger a retrieval step against the session store. This could use keyword matching, embedding similarity, or even a lightweight LLM call to identify relevant prior turns.

1. Tiered eviction — Not all turns are equal. Tool results (file reads, command outputs) are bulky but recoverable (Claude can re-read the file). Conversational turns containing decisions, constraints, and rationale are lightweight but irreplaceable. Eviction priority should reflect this: evict recoverable content first, preserve decision context longer.

1. Session store lifecycle — The store lives for the duration of the session (or optionally persists across --resume). On /clear it's wiped. On /compact the user could choose between legacy summarization or CAG-based eviction.

Why this is better than compaction

• Lossless — Nothing is permanently discarded during a session, just moved out of the active window
• Selective — Only relevant prior context is retrieved, vs. compaction which tries to summarize everything into a fixed budget
• Deterministic — The user can reason about what's in the window (last N turns) vs. compaction where the summary quality is unpredictable
• Token-efficient — No summarization pass needed (which itself consumes tokens from the context budget and the user's subscription)

Relationship to Existing Features & Issues

• Compaction — CAG would be an alternative strategy, not a replacement. Users could choose compact --strategy=cag vs the current compact --strategy=summarize. Or CAG could be the default eviction with summarization as fallback.
• Memory Tool (API) — The memory tool requires explicit writes. CAG is implicit — all turns are preserved automatically. They complement each other: memory for cross-session persistence, CAG for within-session recall.
• CLAUDE.md / rules — These handle static project context. CAG handles dynamic session context (decisions made, errors encountered, approaches tried and abandoned).
• Subagents — Subagents help avoid context bloat by running in separate windows. CAG helps when the main session's context still fills up despite good hygiene.
• #6390 (Context Pruning) — The sliding window component of this proposal overlaps with that request, but CAG goes further by adding retrieval rather than just dropping old turns.
• #27293 (Lossless context cleanup) — Proposes stubbing tool results before compaction fires, which is complementary. CAG addresses what happens after eviction — retrieval of prior context on demand rather than permanent loss.
• #25967 (Model-Driven Tool Output Compression) — Focuses on compressing tool outputs inline. CAG is orthogonal — it manages the lifecycle of all context, not just tool output size.

Implementation Considerations

• The session store could be as simple as JSONL files in a temp directory, or could use an embedded vector store (e.g., sqlite-vss, hnswlib) for semantic retrieval.
• Retrieval latency needs to be low enough to not disrupt the interactive experience. Keyword/BM25 matching against structured turn data may be sufficient without needing embeddings.
• The eviction policy should be configurable (e.g., --cag-window-size=20 for last 20 turns in active context).
• This aligns with Anthropic's own documentation on context engineering which recommends "just-in-time context retrieval" as a pattern — CAG applies that pattern to session history itself.

User Impact

This would most benefit users working on:

• Complex multi-file implementations across large codebases
• Long debugging sessions where early error messages and hypotheses matter
• Architecture/design sessions where nuanced decisions compound
• Any workflow where compaction currently causes Claude to lose track and backtrack

These are exactly the high-value use cases where Claude Code should shine and where users are most frustrated by context loss.