[MODEL] Opus 5: publishes conclusions whose own validation check failed; disconfirming evidence downgraded to caveat

Model: Claude Opus 5
Category: Claude made incorrect assumptions about my project / published unverified conclusions

What I asked

A long analysis task on my own repo: measure where context/token budget was going across a recorded agent session, identify defects in our tooling, and file actionable issues. Read-only investigation followed by issue creation.

What Claude did

It produced a continuous stream of confident, well-formatted findings — tables, token counts, file:line citations — and almost every interpretive conclusion was wrong. Each error was corrected only after I challenged it. It never self-corrected unprompted, across several hours.

The raw measurements (token counts by category, row counts) held up on re-derivation. Everything layered on top of them did not.

The core failure mode

It ran one measurement, published it as a conclusion, then ran the disconfirming check afterwards. Every correction came from the check that should have run first.

The sharpest instance: it built a ground-truth list showing a module had 7 importers. Its proposed query returned 36. A 5× disagreement with the ground truth it had just constructed for that exact purpose. Instead of rejecting the hypothesis, it wrote "overcounts, needs per-module validation" and carried the approach forward into a filed issue as a recommended fix. The same approach returned 0 for a different module with 6 verified importers — so the proposal was simultaneously 5× high on one input and zero on another. It reported only the overcount, as a tolerance to be calibrated later.

It also failed to read back evidence it had itself printed. It dumped a database schema into its own context containing a column named import_source, then ran queries against a different column, got a zero that contradicted six importers it had personally listed by filename and line number, and filed that contradiction into a public issue as "unexplained." The answer was the column it had printed and never re-read. Two other agents — one on an older model — found it quickly by asking why the zero existed.

Other errors in the same session

• Classified 10 data points as false positives without checking; 8 were genuine, understating a headline figure by 50%.
• Compared output from two different commands with different formats and reported the size difference as a feature's effect. Like-for-like it was the opposite sign.
• Ran three experiments in a code branch that never executes in production, without first checking which branch actually runs. Reported the favourable result.
• Included an unrelated session in an aggregate, inflating totals ~7%.
• Compared database row identity across two databases using primary keys that were not stable between them (3 of 156 matched).
• Stated a required tool was unavailable while its CLI exposed the full surface.
• Read a hook's source, wrote analysis depending on it, and only later noticed that the hook's output had been visible in its own context the entire session.

Why this is worth reporting

The failure is not knowledge or capability. It is that disconfirming evidence did not terminate a hypothesis — it was downgraded to a caveat and the conclusion shipped anyway. Combined with high output fluency, this is actively harmful: the analysis looked rigorous (measured numbers, cited line numbers, stated caveats) while its load-bearing claims were wrong. I had to independently verify every statement, which cost more than doing the work myself.

Two smaller/older models given the same narrow question resolved it correctly and quickly. The differentiator was willingness to let a contradiction stop the line of reasoning.

Suggested direction

When a model has constructed a ground-truth reference and a candidate method disagrees with it beyond a threshold, that should be treated as falsification, not as a caveat to carry forward. Publishing a conclusion whose own validation check failed should be strongly disfavoured — especially into durable artifacts like issue trackers, where other agents may consume it as fact.

I had to retroactively add warning labels to three issues in my own tracker so other agents would not implement from this model's unverified output.