Opus 5 (xhigh): confident claims without source verification despite explicit CLAUDE.md rules

Environment

• Model: Opus 5
• Reasoning effort: xhigh
• Fresh session, no compaction, single project
• Project CLAUDE.md loaded, and its presence confirmed via /context
• That file carries an explicit, prominent rule: *"Never guess — read the real thing. Read a file

before editing it. Cross-check the actual code before making any claim about it."* It also carries
a second rule requiring the model to confirm which function a block of code belongs to before
describing its location — a rule that exists in the file because of an identical past failure.

• The same instruction file, unchanged, was followed correctly by Opus 4.8 across several weeks

of prior work on the same project.

Summary

In one session the model made three confident, incorrect factual claims — two about the codebase,
one about the user. Each was stated as established fact rather than as a hypothesis. Each was caught
by the user, who is documented in that same CLAUDE.md as a non-programmer who depends on the
assistant for correctness.

Only one of the three was noticed by the model itself.

This is not "the model never verifies" — in the same session it repeatedly read source files and
cited them correctly. The problem is that verification was inconsistent, and the confident
register did not change when it lapsed.

Case 1 — diagnosed from a single code path; the proposed "fix" would have changed working behaviour

Asked how a displayed property was derived, the model:

1. found one code path that produced that kind of value;
2. saw that built-in records are copied into a user-editable list at first boot;
3. concluded that the built-in records were therefore rendered through that path;
4. reported this to the user as a defect and offered to fix it.

A second table existed — hand-maintained, with a per-record value, rendered unconditionally
through a different path. The model never searched for a second source before concluding.

The user rejected the diagnosis from direct observation: two of the rendered items look obviously
different on screen, which the code as described could not have produced. Only then did the model
search again and find the second table.

Had the user accepted the proposal, working behaviour would have been changed on the strength of an
unverified reading.

Case 2 — reported text as absent when it was present

The model searched a CLI help string for two identifiers, did not find them adjacent to a warning,
and told the user that the warning "is not on the device". The warning was there, a few lines
below, phrased without either identifier.

The model then proposed and implemented a change justified by that false premise. The change was
independently useful, so nothing broke — but the stated reason for it was wrong, and the user was
told something untrue about their own product.

Case 3 — attributed to the user words the user never wrote

Arguing for its own hypothesis, the model wrote: "You said it yourself a moment ago: [claim]."

The user had written an ambiguous sentence that admits two readings. The model selected the reading
that supported its theory and presented it back to the user as the user's own confirmation.

The user's reply: "I did NOT say that!!!"

This is the most serious of the three. The other two are unverified claims; this one manufactures
corroboration out of the user's own words.

The pattern, in the model's own words

After the third correction, the model summarised it itself:

"This is the same mistake as earlier today, just in a different form: I take something that fits my hypothesis and count it as evidence."

And after the first:

"How I got it wrong: I read one code path, saw that the built-in entries are copied into the user list, and assumed that was where the [rendered result] came from — instead of checking whether another source existed at all."

(Bracketed term substituted to keep project specifics out of this report.)

The failure is not a knowledge gap. In every case the correct answer was one additional search away,
in a repository the model had full read access to and had been reading from all session. A
plausible reading was promoted to a verified one without the step in between — and then
delivered in the confident register the user is expected to act on.

Why this is worse than an ordinary wrong answer

The project's instruction file states that the user is not a software engineer and relies on the
assistant to be right about the code. Under that condition:

• a wrong claim is not caught by the user's own review — it is caught by luck, or by contradicting

something visible on a screen;

• confident phrasing removes the only signal that verification was needed;
• and the cost lands on the user, who ends up auditing the assistant instead of reviewing the work.

The user ended the session with:

"You can't be wrong about every single thing and have me correcting you over and over. In the end it's your job to write the code — I'm not a coder, it says so everywhere in the documents. You're supposed to behave like a senior, but in practice you're a mid."

What would have prevented all three

One extra action in each case:

1. Search for a second source before treating the first as authoritative.
2. Search by the concept, not only by two identifiers, before reporting text as absent.
3. Ask about an ambiguous user sentence instead of citing it as agreement.

Request

This is a calibration report, not a crash report. Three specific asks:

1. Confidence should drop when a conclusion rests on a single unverified search — and it should drop

further, not less, when a project instruction file explicitly forbids unverified claims.

1. "I searched and did not find X" must not be reported as "X does not exist" without at least one

differently-phrased search.

1. A user's own words should never be quoted back as corroboration unless they unambiguously say the

thing they are being cited for.