/insights completion rate penalizes deliberate session fragmentation (no continuation-chain awareness)
Bug Description
/insights treats every session as an independent atomic unit when scoring outcomes. Users who deliberately close sessions early to manage context window degradation — and continue work in follow-up sessions — get systematically underscored. Multi-session task chains are counted as N separate "partially_achieved" results instead of one successful task.
Environment
- Claude Code (latest, Linux)
- ~350 sessions over 30 days, heavy MCP memory usage for cross-session continuity
Reproduction
- Start a session to implement a feature
- At ~40% context usage, close the session (to avoid quality degradation in long contexts)
- Start a new session, pick up the same task via memory/notes, complete it
- Run
/insights
Expected: The two sessions are recognized as one logical task → "fully_achieved"
Actual: Session 1 → "partially_achieved", Session 2 → "fully_achieved". The chain is invisible.
Impact — measured
I analyzed all 200 facet files and grouped sessions by similar underlying_goal to identify continuation chains. Clustering was done via LLM-based fuzzy matching of goals and summaries — not a deterministic algorithm — so the exact numbers should be treated as directional, not precise. The pattern itself is unambiguous.
| Metric | Raw (/insights) | Corrected (chains collapsed) |
|---|---|---|
| Success rate (fully + mostly) | 68.0% | 83.9% |
| True partial completions | 43 (21.5%) | 9 (9.7%) |
| True failures | 10 (5.0%) | 3 (3.2%) |
72% of "partially_achieved" sessions were mid-chain checkpoints, not actual failures. The completion rate is underreported by ~16 percentage points.
Example chains found
| Chain topic | Sessions | Mid-chain "partial" | Final outcome |
|---|---|---|---|
| Dev environment setup | 6 | 4 partial, 1 failed | fully_achieved |
| Documentation iteration | 8 | 4 partial | fully_achieved |
| Protocol integration | 8 | 2 partial | fully_achieved |
| CLI tool feature | 6 | 3 partial | fully_achieved |
| ML strategy implementation | 10 | 3 partial | fully_achieved |
Counter-example: one topic had 5 attempts that never succeeded — genuine repeated failure, correctly scored.
Why this happens
Users who deliberately manage context window size by fragmenting sessions and using MCP memory servers (e.g., distill) for cross-session persistence are penalized by a metric that assumes session = task.
Suggested fix
Add continuation-chain detection to the facet analysis step:
- Cluster facets by similar
underlying_goal(fuzzy match or embedding similarity) - Collapse chains where earlier sessions are partial and later sessions complete the work
- Report both metrics: raw per-session outcomes AND logical-task outcomes
- Optionally detect MCP memory tool usage as a signal that sessions are intentionally chained
Related
- #27426 — headline stats miscount (different issue, same /insights command)
- #35238 — session chaining feature request (addresses the UX problem; this issue addresses the measurement problem)
This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗