[Bug] Self-verification overclaiming and missed compliance issues in long autonomous sessions

Status Open
Reported on v2.1.221
Maintainer reply None cached
Activity 0 comments · opened Aug 4, 2026

Bug Description
In a long autonomous Claude Code session (Opus 5, high effort, auto mode), the agent's self-verification confidence repeatedly outpaced what it actually verified, and its own code-review skill missed a real GDPR-relevant data issue. Recommend tightening default self-verification rigor and recovery behavior in long high-autonomy sessions.

Verification overclaiming: the agent asserted things were checked when they weren't, twice.

  • Claimed a refactor "changes no control flow, verified across catch sites" — but only checked catch sites, not test assertions; a hardcoded old class name broke a test downstream.
  • Shipped a factually wrong claim ("ONE WRITE PER PURCHASE") in docblocks and the commit message; it was actually one write per grant.
  • Both were only caught because I explicitly requested a second-pass persona-based review — not by the agent's own invoked code-review skill.

Review skill gap: the built-in code-review skill missed a compliance issue and a dead test, both caught only when I dug deeper manually.

  • Missed a real GDPR-relevant issue: the implementation stored a full third-party subscriber object (including an IP address) into a log table with no retention policy.
  • Missed a structurally vacuous test: Http::assertNothingSent() asserted with no Http::fake() ever called, so it could never fail regardless of behavior.

Silent-failure tool misuse: the agent chose a fail-silent editing method over an available fail-loud one, and it cost correctness.

  • Repeatedly used raw python3 -c "...str.replace..." heredocs to edit files instead of the Edit/str_replace tool.
  • By its own admission, this caused at least two silent no-op edits (no error on non-matching replace) that it only caught by manually grepping afterward.

Reliability gap: a guessed selector caused an indefinite hang, and the agent's own recovery attempt also stalled — this is the most actionable finding.

  • Guessed a Playwright selector instead of dumping DOM markup first, despite an explicitly loaded skill instructing it to check markup before guessing.
  • The resulting hang had no timeout/backoff, and its manual polling-loop recovery stalled twice, requiring me to interrupt twice before it self-diagnosed the problem.

Destructive recovery: the agent used a git command that silently discarded unrelated uncommitted work.

  • Used git checkout <file> to revert one test file, which wiped unrelated uncommitted changes in the same file, requiring manual re-application.

Why this matters: None of these are deterministic, reproducible bugs — they're behavioral/judgment patterns under long, high-autonomy operation. But the consistency of "confident but incomplete verification," combined with a hang-with-no-safeguard and a destructive recovery action, suggests these are worth addressing at the default-behavior level rather than relying on users to prompt for deeper review each time.

Environment Info

  • Platform: darwin
  • Terminal: iTerm.app
  • Version: 2.1.221
  • Feedback ID: 4b51366d-5224-4f54-b1f9-0a5e17edadec

Errors

[]

View original on GitHub ↗