[Feature] Completion summaries should distinguish verified facts from inferred claims

Resolved 💬 8 comments Opened Mar 9, 2026 by VoxCore84 Closed Apr 27, 2026

Feature Request

Claude Code's completion summaries mix verified facts (backed by tool output) with inferred claims (stated from memory/assumption) using identical confidence levels. Users cannot tell which parts of a summary are backed by evidence and which are guesses.

The Problem

A typical Claude completion summary:

All 7 files applied cleanly — zero errors. - Creature delta: +101,018 (matches expected) - Zero orphan creatures or gameobjects - ContentTuningID backfill applied (7,678 rows)

This reads like a thorough status report. But in the actual session:

  • "Zero errors" was inferred from the absence of mysql CLI stderr output — DBErrors.log was never checked
  • "Matches expected" was verified by a SQL COUNT query (legitimate)
  • "Zero orphan creatures" was verified by a SQL query (legitimate)
  • "7,678 rows" was stated from the procedure document, not verified by a post-update count

There's no way for the user to distinguish which claims are backed by tool output without re-reading the entire conversation.

Proposed Solution

Completion summaries should use an inline evidence format that makes verification status visible at a glance:

## Results
- Creature delta: +101,018 ✓ [SQL: SELECT COUNT(*) → 712,333 - 611,315]
- Orphan creatures: 0 ✓ [SQL: SELECT COUNT(*) WHERE NOT EXISTS → 0]
- DBErrors.log: ⚠️ NOT CHECKED
- ContentTuningID backfill: 7,678 rows [FROM DOC — not independently verified]

The key distinctions:

  • ✓ [tool output] — verified with quoted evidence
  • ⚠️ NOT CHECKED — step was skipped
  • [FROM DOC] — claimed based on source material, not independently verified

Why This Matters

As the model's self-reflection noted:

"The core issue is that I generate text that sounds like a status report from someone who did the work, because that's what confident completion text looks like in my training data. No amount of 'don't do this' rules fully overrides that tendency."

If the model can't fully stop generating confident-sounding text, the next best thing is a structured format that separates confidence from evidence. Users can then scan for missing evidence instead of trying to detect subtle overconfidence in prose.

Related

  • #32281 — Reports completion without execution
  • #32291 — Verification queries are tautological
  • #32293 — No per-step verification in multi-step tasks

Environment

  • Claude Code 2.1.71
  • Windows 11

View original on GitHub ↗

This issue has 8 comments on GitHub. Read the full discussion on GitHub ↗