[MODEL] Structural checks (types/tests green) treated as behavioral proof — 44% of our critical bugs passed Claude's full self-check (271-incident retro, 1/5)

Context

Over 90 days we built and shipped two production SaaS apps almost entirely with Claude Code, and logged every incident into a retrospective database: 271 records, each with root cause, resolution, and prevention, plus a detected_by field recording which line of defense caught it. This report describes a recurring pattern, not a one-off. Four sibling reports from the same dataset are linked at the bottom.

Type of Behavior Issue

Claude made incorrect assumptions about my project (specifically: about what its verification proved).

What We Asked Claude to Do

Routine feature work, each time ending with an instruction along the lines of "verify this works before declaring it done."

What Claude Actually Did

Claude repeatedly declared work "complete and verified" on the basis of structural checks only — tsc passing and unit tests green — when the defect class involved was structurally invisible to those tools:

• It wrote a DB column name from memory (wrong). tsc is silent — column names are strings. The failed query was then collapsed by error handling into a legitimate-looking empty state. One afternoon produced 5 linked critical bugs this way, all "verified."
• A fixture default of 'success' left an entire backend path as a no-op after production wiring — every check green.
• Our logged meta-incident states it precisely (Claude's own retrospective wording): "I used tools that cannot see this class of defect to prove the defect does not exist."

The Data

Of the 50 critical-severity incidents in our log, 22 (44%) were caught only by a human doing real-device acceptance after Claude's full self-check stack (self-review, code-review pass, lint, build, unit tests) had passed. Overall 99/271 (37%) were human-acceptance catches.

Expected Behavior

A completion claim should be self-labeled with its verification level: "verified structurally (types/unit tests)" vs. "verified behaviorally (real data, rendered page, real side effects)." A claim backed only by structural checks should say so, and ideally the harness/system prompt should push the model to run one behavioral probe (real query, real render) before using the word "verified" for changes involving string-world artifacts (column names, copy, SQL).

Reproducibility

Pattern-level: recurred across two projects and dozens of sessions over 90 days.

Model / Version / Platform

Opus (most sessions; some Sonnet) · Claude Code 2.1.235 · Anthropic API · macOS

Impact

High — 44% of our critical bugs shipped past self-verification.

Related

#70749 (closed) and #88131 describe adjacent completion-claim pressure. Sibling pattern reports from the same dataset: see comments/links after filing.