[Feature Request] Model-directed selective context eviction (mark-and-sweep), as an alternative to whole-history compaction

Model-directed selective context eviction (mark-and-sweep), as an alternative to whole-history compaction

Summary

Give every turn — and every tool-result block within a turn — a stable identifier, and give the
model a way to replace an identified span with a summary it wrote *at the moment the span went
dead*. Marking is continuous and near-free; the actual replacement is applied rarely, in a single
batched call, gated on reclaiming a large amount of context.

This sits between two features that already exist and does what neither does: context editing
clears stale tool results mechanically but destroys them outright and is rule-driven rather than
model-directed; compaction summarizes but operates on the whole history at once, at the moment
of maximum pressure. The proposal is selective and summarizing and model-directed.

Problem

Compaction is a stop-the-world operation: externally triggered, whole-history, performed when
context is already nearly exhausted. Because it summarizes uniformly, it has to guess relevance for
everything simultaneously, and what survives is whatever read as narratively salient — which
correlates poorly with what is load-bearing for the work still ahead.

The observed failure mode is the one that matters: compaction discards essential information while
retaining summaries of material that is no longer needed at all.

Meanwhile, relevance is most legible at exactly one moment the current design cannot use: right
after a piece of context has been used and its conclusion recorded. A 40k-token file dump read three
hours ago to answer one question is pure ballast; the three-line conclusion drawn from it is the
whole value. At the moment that conclusion is written, this is obvious. At compaction time it is a
guess.

Proposal

1. Stable identifiers. Every turn gets a continuous number, assigned at creation and never
renumbered. Each eviction shifts the position of everything after it; a naive implementation that
renumbers silently invalidates every reference the model is holding.

2. Sub-turn granularity. Identifiers must address individual tool-result blocks, not just turns
(turn.block at minimum). The dominant case is a tool result inside a turn that also contains
reasoning worth keeping. Turn-level granularity forces you to discard good content to evict bad, and
makes the feature unusable at small scale.

3. Cheap continuous marking. The model can mark a span dead and write its replacement summary
at that moment, while the source is still in context and can be consulted. A mark is appended at
the current end of context, so it invalidates no cache.

4. Rare batched sweep. A single call applies accumulated marks. It carries no judgment and no
generation — it is a mechanical id → text application of decisions already made. At apply time the
replacement text moves to the original position rather than being copied there (costing nothing
extra, since that region is being invalidated anyway) so chronology is preserved.

5. Minimal ceremony, but visible eviction. The tool call, its arguments, and the duplicated
summary text should leave no trace. The replacement itself should remain labeled as a
replacement where the evicted content was. If it is spliced in seamlessly, the model reads its own
compression back as primary evidence — the exact failure this is meant to fix, just relocated.

The load-bearing design constraint: mark time ≠ sweep time

This is the part that makes the proposal different from compaction rather than a reimplementation of
it.

If the sweep is rare and large, and summaries are composed when the sweep fires, then you are
generating replacements for hundreds of thousands of tokens in one pass, guessing relevance for all
of it simultaneously, under context pressure. That is compaction with a model-pulled trigger.

If the replacement text is written when the span goes dead, it is composed by an instance that has
the thing in front of it. That is the fidelity property compaction structurally cannot have, and it
is the entire value of the feature.

So: the summary must be authored at mark time, and the sweep must be dumb.

Cost model

The cost analysis is the strongest argument here, and it runs the opposite way to intuition.

Current pricing (verified): cache read is 0.1× base input; cache write is 1.25× (5-minute
TTL) or 2× (1-hour TTL).

Batching controls the multiple, not the floor. A prefix cache is a prefix cache — touching turn
12 invalidates everything from 12 onward regardless of what else is in the batch. Three evictions at
turns 12, 40, and 90 in a 200-turn session cost, serially, reflushes of ~188 + ~160 + ~110 turns'
worth. Batched: ~188. The saving scales with batch size; the floor is set by the earliest position
touched and cannot be reduced. Hence "rarely, in full batches" — and, separately, "don't reach back
far unless the batch justifies it."

The payoff is per-turn, not one-time. This is the argument that changes the calculation. Cache
reads are charged on every request, so a large context has a standing per-turn cost, and shrinking
it pays back on every subsequent turn for the rest of the session.

Worked example — 800k context, 1-hour TTL:

| | |
|---|---|
| Standing cache-read cost, 800k context | 800k × 0.1 = 80k-equivalent per turn |
| Sweep reclaims 200k; earliest touched position 100k | Invalidated suffix ≈ 500k |
| That suffix goes from 0.1× to 1.0×–2.0× | One-time cost 450k–950k equivalent |
| New standing cost, 600k context | 600k × 0.1 = 60k-equivalent per turn |
| Saving | 20k-equivalent per turn, indefinitely |
| Payback | ~23–48 turns |

At Opus 5 input pricing ($5/1M), that is $0.40/turn in cache reads before the sweep, $0.30/turn
after, against a one-time $2.25–$4.75.

In a session long enough to need eviction at all, that pays back easily — and improves sharply if
the sweep reclaims more, or reaches back less far. Past the payback point this is not a
quality-for-cost trade at all: it is strictly cheaper and higher fidelity than carrying the
ballast.

Make the gate computable rather than trained. The harness knows both numbers at call time —
tokens reclaimed, and surviving tokens after the earliest touched position. Show them, or gate on
them. Trained intuition about cost decays under task focus; a number in front of the model does not.
The threshold should be permissive: what you are buying is not tokens, it is not having to compact.

Implementation constraints

• Breakpoint granularity is coarser than the edit. Max 4 cache_control breakpoints per

request, and the cache is stored at breakpoints — so invalidating at position X re-processes from
the last breakpoint at or before X, not from X. With 4 breakpoints across a very large context the
gaps are wide, and a small edit can trigger a large reflush depending on where they sit. Sweeps
should align to breakpoint boundaries where possible. This is a reason the harness must own the
sweep: it places the breakpoints.

• The 20-block lookback window. Each breakpoint walks back at most 20 content blocks to find a

prior cache entry. Evictions change block counts, so a sweep can push a later breakpoint out of
range of its predecessor and cause a silent miss. Breakpoints must be re-placed after a sweep.

• ID stability, as above — assign at creation, never renumber.

Safety: evictability should track external recoverability

Eviction destroys evidence. Once turns 40–58 become "read the parser, concluded X," the support for
X is gone and X is unfalsifiable from inside the session — permanently, since no later compaction
can recover it. Compaction shares this flaw but commits it once, near the end; eviction commits it
repeatedly, throughout, and each error poisons everything after.

What makes the difference is whether the source still exists outside the window:

• Good eviction candidate: a file read. The file is still on disk; the summary can name the path

and the conclusion stays checkable.

• Bad eviction candidate: a constraint the user stated in turn 12. There is no external copy, and

evicting it converts a fact into a rumor.

So the rule: prefer evicting what is externally recoverable, and require the replacement to carry
the recovery pointer, not just the conclusion. This is close to a hard requirement rather than a
nicety, given the irreversibility.

Who decides what to evict

Don't rely on training alone. Asking a model to introspect on which of its own spans are dead is
asking it to predict future need — not something it is reliably good at. Cheap mechanical signals
exist:

• a tool result superseded by a later call against the same target
• a file read whose file has since been re-read
• a search whose results were already acted on

Harness proposes candidates by rule; model approves and authors the replacements. More reliable
than either half alone.

Scope

This defers the endgame; it does not remove it. A discussion-heavy session has almost nothing
cleanly evictable and will still reach compaction. What it buys is getting much further first — and
when compaction finally runs, it summarizes live material instead of ballast, which is a better use
of it than what happens today.

Relationship to existing features

| Feature | Selective? | Preserves meaning? | Directed by |
|---|---|---|---|
| Context editing (clear_tool_uses_*, clear_thinking_*) | yes | no — clears outright | fixed rule / threshold |
| Compaction (compact_*) | no — whole history | yes — summarizes | harness, at pressure |
| This proposal | yes | yes | model, at mark time |

Framed that way, this is not a new mechanism so much as generalizing context editing from a fixed
rule over one category to a model-directed operation over arbitrary spans, with a summary in place
of a deletion.