/insights completion rate penalizes deliberate session fragmentation (no continuation-chain awareness)

Resolved 💬 2 comments Opened Mar 22, 2026 by 5queezer Closed Apr 19, 2026

Bug Description

/insights treats every session as an independent atomic unit when scoring outcomes. Users who deliberately close sessions early to manage context window degradation — and continue work in follow-up sessions — get systematically underscored. Multi-session task chains are counted as N separate "partially_achieved" results instead of one successful task.

Environment

  • Claude Code (latest, Linux)
  • ~350 sessions over 30 days, heavy MCP memory usage for cross-session continuity

Reproduction

  1. Start a session to implement a feature
  2. At ~40% context usage, close the session (to avoid quality degradation in long contexts)
  3. Start a new session, pick up the same task via memory/notes, complete it
  4. Run /insights

Expected: The two sessions are recognized as one logical task → "fully_achieved"
Actual: Session 1 → "partially_achieved", Session 2 → "fully_achieved". The chain is invisible.

Impact — measured

I analyzed all 200 facet files and grouped sessions by similar underlying_goal to identify continuation chains. Clustering was done via LLM-based fuzzy matching of goals and summaries — not a deterministic algorithm — so the exact numbers should be treated as directional, not precise. The pattern itself is unambiguous.

| Metric | Raw (/insights) | Corrected (chains collapsed) |
|---|---|---|
| Success rate (fully + mostly) | 68.0% | 83.9% |
| True partial completions | 43 (21.5%) | 9 (9.7%) |
| True failures | 10 (5.0%) | 3 (3.2%) |

72% of "partially_achieved" sessions were mid-chain checkpoints, not actual failures. The completion rate is underreported by ~16 percentage points.

Example chains found

| Chain topic | Sessions | Mid-chain "partial" | Final outcome |
|---|---|---|---|
| Dev environment setup | 6 | 4 partial, 1 failed | fully_achieved |
| Documentation iteration | 8 | 4 partial | fully_achieved |
| Protocol integration | 8 | 2 partial | fully_achieved |
| CLI tool feature | 6 | 3 partial | fully_achieved |
| ML strategy implementation | 10 | 3 partial | fully_achieved |

Counter-example: one topic had 5 attempts that never succeeded — genuine repeated failure, correctly scored.

Why this happens

Users who deliberately manage context window size by fragmenting sessions and using MCP memory servers (e.g., distill) for cross-session persistence are penalized by a metric that assumes session = task.

Suggested fix

Add continuation-chain detection to the facet analysis step:

  1. Cluster facets by similar underlying_goal (fuzzy match or embedding similarity)
  2. Collapse chains where earlier sessions are partial and later sessions complete the work
  3. Report both metrics: raw per-session outcomes AND logical-task outcomes
  4. Optionally detect MCP memory tool usage as a signal that sessions are intentionally chained

Related

  • #27426 — headline stats miscount (different issue, same /insights command)
  • #35238 — session chaining feature request (addresses the UX problem; this issue addresses the measurement problem)

View original on GitHub ↗

This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗