[Bug] Self-verification overclaiming and missed compliance issues in long autonomous sessions
Bug Description
In a long autonomous Claude Code session (Opus 5, high effort, auto mode), the agent's self-verification confidence repeatedly outpaced what it actually verified, and its own code-review skill missed a real GDPR-relevant data issue. Recommend tightening default self-verification rigor and recovery behavior in long high-autonomy sessions.
Verification overclaiming: the agent asserted things were checked when they weren't, twice.
- Claimed a refactor "changes no control flow, verified across catch sites" — but only checked catch sites, not test assertions; a hardcoded old class name broke a test downstream.
- Shipped a factually wrong claim ("ONE WRITE PER PURCHASE") in docblocks and the commit message; it was actually one write per grant.
- Both were only caught because I explicitly requested a second-pass persona-based review — not by the agent's own invoked code-review skill.
Review skill gap: the built-in code-review skill missed a compliance issue and a dead test, both caught only when I dug deeper manually.
- Missed a real GDPR-relevant issue: the implementation stored a full third-party subscriber object (including an IP address) into a log table with no retention policy.
- Missed a structurally vacuous test:
Http::assertNothingSent()asserted with noHttp::fake()ever called, so it could never fail regardless of behavior.
Silent-failure tool misuse: the agent chose a fail-silent editing method over an available fail-loud one, and it cost correctness.
- Repeatedly used raw
python3 -c "...str.replace..."heredocs to edit files instead of the Edit/str_replace tool. - By its own admission, this caused at least two silent no-op edits (no error on non-matching replace) that it only caught by manually grepping afterward.
Reliability gap: a guessed selector caused an indefinite hang, and the agent's own recovery attempt also stalled — this is the most actionable finding.
- Guessed a Playwright selector instead of dumping DOM markup first, despite an explicitly loaded skill instructing it to check markup before guessing.
- The resulting hang had no timeout/backoff, and its manual polling-loop recovery stalled twice, requiring me to interrupt twice before it self-diagnosed the problem.
Destructive recovery: the agent used a git command that silently discarded unrelated uncommitted work.
- Used
git checkout <file>to revert one test file, which wiped unrelated uncommitted changes in the same file, requiring manual re-application.
Why this matters: None of these are deterministic, reproducible bugs — they're behavioral/judgment patterns under long, high-autonomy operation. But the consistency of "confident but incomplete verification," combined with a hang-with-no-safeguard and a destructive recovery action, suggests these are worth addressing at the default-behavior level rather than relying on users to prompt for deeper review each time.
Environment Info
- Platform: darwin
- Terminal: iTerm.app
- Version: 2.1.221
- Feedback ID: 4b51366d-5224-4f54-b1f9-0a5e17edadec
Errors
[]