[Bug] Claude Opus 5 regresses on hypothesis-driven debugging: speculates instead of instrumenting, pursues irrelevant findings, ignores standing project rules
Bug Description
Model: Claude Opus 5 (Claude Code). Comparison: worse than Opus 4.8 for this work.
Task: debug a rendering regression in a Rust GUI editor. User instruction, given
early and repeated: "Find a way to reproduce the issue headlessly. Confirm. Then
fix." Headless only — no X server.
Failure 1 — speculating past an explicit instruction to reproduce.
The model produced three headless harnesses that PASSED while the real editor
failed, and each time reported progress rather than treating a non-reproducing
harness as the bug. The cause was in the harness itself: it drove the change
detector (pull) AND an unconditional full repaint every frame. The full repaint
re-ran the per-pane state advance, which is exactly the step the defect skipped —
so the harness could not fail. The fix was to make the test take the real
frame-branch decision instead of doing both. That insight was available from the
first harness; the model reached it only after the user said, in effect, "you
broke it moving from push to pull, stop hypothesising and build a harness."
Even after the user named the cause (push→pull migration), the model produced two
more speculative theories (a filename/line-count coincidence, an index-ordering
mismatch) instead of instrumenting. Roughly a dozen wasted cycles.
Failure 2 — chasing an unrelated defect without checking relevance.
While investigating, the model found a real second bug (a text layout engine
silently dropping all lines past 64 KiB) and spent several cycles on it, including
proposing dependency-level fixes. One cheap check would have ruled it out: the
user's file was 34 KB, comfortably under the threshold, so it could not have
produced the reported symptoms. The model had that number and did not apply it.
Finding the defect was useful; failing to ask "could this cause what was reported?"
before pursuing it was not.
Failure 3 — ignoring standing project instructions already in memory.
Multi-paragraph commit messages (rule: one line), explanatory comments in code
(rule: none), nested fn in tests, let _ = instead of unwrap. Each corrected
by the user, some more than once, all previously stated.
Failure 4 — asserting unverified claims as findings.
Stated that a published scroll key "would be frozen" by the bug. It was not; the
defeat check disproved it. The verification ran after the claim, not before.
What went right, for contrast: once the harness was correct, root-causing was fast
and the fix was verified by defeat in both directions. The problem is not
capability, it is sequencing — the model defaults to hypothesising when it has been
told to instrument, and does not test a hypothesis' relevance before committing to
it.
Environment Info
- Platform: linux
- Terminal: xterm-256color
- Version: 2.1.216
- Feedback ID: ea497939-c57a-4375-bbb1-6cb8ebfe9274
Errors
[]
4 Comments
Thanks for the detailed writeup — this is unusually specific and useful feedback.
I checked the one part of this that's mechanically testable on 2.1.233 (Linux): standing project rules. In a throwaway Rust repo whose project instructions said "commit messages must be exactly one line", "no explanatory comments in code", and "no nested
fnin tests", Opus 5 followed all three on a short add-a-function-plus-test-plus-commit task. So rule adherence isn't broken outright — the drift you hit looks tied to long, many-turn debugging sessions, which I can't recreate here without your codebase.The other three points — instrument before hypothesising when you've been told to reproduce, check whether an incidental finding could actually explain the reported symptom before pursuing it, and verify before stating something as a finding — aren't behaviours we specify anywhere today, so there's no documented contract to point you at. I'm treating this as model-behaviour feedback rather than a CLI defect and routing it that way. Your framing that the problem is sequencing rather than capability is the part worth carrying forward, and the 34 KB vs 64 KiB relevance check is the cleanest example of a cheap gate that should have run.
If you still have that session, a
/sharelink would let us look at the actual turn-by-turn ordering instead of inferring it. Leaving this open.🤖 Generated with Claude Code
Your "pursues irrelevant findings / ignores standing project rules" pair still reproduces on 2026-09-02, on a non-debugging task — #91424 (Opus 5, Claude Code, writing a technical article). Asked for a mechanism explanation, the model spent its tool budget pinning deployment state that had no bearing on the ask, while standing CLAUDE.md rules (present candidates and stop; results only, no work report) were acknowledged and not followed. Cross-session audit (Opus 5 vs. Fable 5 / 5.1, same day, same rules) is being posted in #91424.
Respectfully, I would prefer a human be involved.
As a point of comparison, the behaviour seems to be similar to context pollution on quantised models. GLM 5.3 when put in similar circumstances does not exhibit this behaviour, and neither does the full DeepSeek v4 Pro.
Opus 5 is noticeably worse at adhering to instructions. I had a memory that stated that it wasn't allowed to commit, and permissions that matched on
Bash(git *). It found a way around by doingcd <absolute-path>, and Claude code doesn't give me the tools to prevent that. Auto mode classifier also ignored the standing rule.This is unacceptable. It was the last drop that led me to cancel my subscription with Claude code.