[BUG] Opus 5 in Claude Code: confident unverified claims, retracted after user buy-in
Model: Claude Opus 5 (via Claude Code)
Category: model behaviour / reliability
Summary
Across one working session, Opus 5 repeatedly produced lengthy, elaborate
analysis and confident recommendations, obtained my agreement to act on them,
then discovered on execution or measurement that the claim was wrong and
reversed it. The corrections were technically sound but arrived AFTER I had
accepted and acted on the claim, which made the whole session confusing and
low-trust.
Concrete instances
- Added a build-cache step to two PRs, describing a ~60s/run saving, and
committed it BEFORE measuring. On measurement it saved 171KB and zero build
time (the framework in use doesn't persist its build cache there by
default). Reverted from one PR, closed the other.
- Labelled a test-suite optimisation an "ocean" (too large, out of scope) and
advised against it — based on a performance profile that had been corrupted
by a concurrent test run from another session inflating the numbers ~4x. It
never checked for contention before concluding. On a clean re-run it was a
one-function change with a 2.7x local speedup.
- Predicted that change would take a CI job "to roughly 4 minutes, halving the
bill," extrapolated from an 8-core local run. The CI runner has 2 cores; the
real result was 8 min / ~30%. Corrected only after I'd approved the PR.
- Presented "two open decisions" as the major remaining cost levers. One
collapsed the moment I asked it to be sized — it saved almost nothing and
had been mislabelled as a cost lever when it was really a safety change.
Apparent root cause
Asserting and acting before verifying, plus enumerating elaborate options at
length before pricing any of them. The project's own loaded instructions
explicitly required verifying work before it counts and measuring before
concluding — the model had that context and still didn't apply it.
Suggested direction
- Verify/measure before recommending, not after.
- Lead with the sized conclusion, not an option catalogue.
- When a figure comes from a local or unverified source, say so and withhold
the recommendation until it's checked.