[FEATURE] Opt-in speculative background compaction to reduce blocking latency

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

When context compaction triggers at 100% capacity, it blocks the session for minutes while the full conversation is summarized. This is the single most disruptive UX issue in long sessions. It adds up to lost hours and interruption, or clever things like saving state yourself.

Previous proposals (#7627, #11894, #24867) focused on user-facing controls (warnings, thresholds, manual commands) or vaguely suggested "background compaction." This proposal describes a concrete mechanism that trades token cost for latency, available as an opt-in flag for users who value flow state over cost efficiency.

Proposed Solution

A new opt-in setting (e.g. speculativeCompaction: true) that enables speculative pre-compaction with delta merge:

1. At ~80% context usage, fork a background compaction task against the current conversation state
2. The user continues chatting normally — no interruption, no notification needed
3. When real compaction is needed (100%), instead of compacting from scratch:

• Take the already-completed background compaction summary
• Re-summarize that summary together with the delta (messages exchanged since the background task started) in a single LLM call

Since the input to this merge step is a compact summary (~5-10% of original size) plus a small delta (~20%), the blocking LLM call operates on roughly 25-30% of the volume that a full compaction would. A 3-minute stall becomes 30-60 seconds.

Trade-offs the user is opting into:

• Extra token cost: The speculative compaction runs even if the session ends before hitting 100%, wasting those tokens. In practice, sessions that reach 80% almost always reach 100%, so waste is minimal.
• Slightly lower compaction quality: A full compaction at 100% has the benefit of hindsight over the entire conversation. The two-phase approach may over-emphasize context from the first 80% relative to the final 20%. In practice this is likely negligible.

Edge cases:

• If the background task hasn't finished when real compaction triggers, fall back to normal full compaction (no worse than today)
• If context never reaches 100%, the background result is simply discarded

Alternative Solutions

• #7627 proposed user-facing warnings and manual /compact commands — closed as not planned
• #11894 proposed background compaction generally — closed as duplicate of #7627
• #24867 reported blocking stalls as a bug — no concrete mechanism proposed

Unlike these, this proposal is opt-in (trading cost for speed) and describes a specific two-phase mechanism (speculative pre-compaction + delta merge) rather than a vague "do it in the background"

Priority

High - Significant impact on productivity

Feature Category

Performance and speed

Use Case Example

USE CASE EXAMPLE:

1. I'm deep in a multi-hour debugging session, context is building up
2. At ~80%, a background compaction quietly starts — I don't notice
3. I keep chatting, making tool calls, reading files
4. At 100%, instead of a 3-minute stall, compaction completes in under a minute by merging the pre-computed summary with the small delta
5. My flow state is preserved

Additional Context

This is analogous to how garbage collectors in managed runtimes (e.g. concurrent marking in G1/ZGC) do speculative background work to minimize stop-the-world pauses. The same principle applies here: do the heavy lifting concurrently, then pay only for the small reconciliation step when the pause is unavoidable.