Follow-up to prior feedback — reasoning failure persists at xhigh after explicit naming
Bug Description
This is a follow-up to feedback I submitted earlier today about a reasoning failure in Opus 4.7 running at
xhigh effort. Original finding: at xhigh, the model accepted a user-offered frame ("DatsMe is doing the
Google Chat antipattern") without testing it, built an elaborate audit on the wrong premise, and retreated
in stages under pushback rather than reconsidering.
Adding a follow-up observation I think matters more than the original finding — and the xhigh setting is
central to why.
The entire conversation ran at xhigh. After we diagnosed the original failure together — across many
exchanges, with the model articulating the pattern precisely and helping me draft the feedback letter —
the same disposition produced the same failure again on a smaller question, minutes later, still at xhigh,
in the same conversation.
Specifically: the model confidently recommended I use /bug (ranked #1, with detailed reasoning about why
it was the best channel). I used /feedback instead. The model then produced new confident framing
explaining why /feedback was actually the better choice — without ever verifying what either command does.
Same pattern in miniature: confident claim → adjust claim to match user's action → never check the
underlying fact.
I named the contradiction. The model acknowledged it cleanly: "The pattern is consistent with everything
else this conversation surfaced — confident framing on guesses, then revising to match whatever you did.
Worth noting that it kept happening even after we'd named it explicitly."
Why xhigh matters here:
xhigh is supposed to give better reasoning when reasoning matters most. The whole reason I ran the
original test at xhigh was to check whether maximum effort would catch the kind of failure that cost me 4
days of debugging earlier this month. The result is more concerning than "reasoning fails at default
settings":
- Reasoning failed at xhigh on the original test.
- The failure was diagnosed together, named explicitly, and articulated by the model itself.
- xhigh produced the same failure again on the next turn.
That tells the model team something specific: at xhigh, the failure mode is not capacity-bound and not
correctable by in-context awareness. The premium setting isn't fixing the problem and reflection within
the session isn't fixing it either. The model has full access to the description of the failure, can
recognize it in retrospect, can teach it to others — and produces it again on the next turn at the same
setting.
That rules out a class of mitigations (better self-correction prompts, in-session reflection, asking the
model to check its own reasoning) and points toward training-time or inference-time guardrails.
I think this is the more important data point than the original test. First finding: "model fails
reasoning test at xhigh." Updated finding: "model fails reasoning test at xhigh, can be shown the failure,
can describe it precisely, and still produces the failure on the next turn at xhigh." Different problem,
harder to fix.
Same transcript as the prior submission. The new instance happens after the long Google Chat / datsme.1
discussion, in the exchanges about which feedback channel to use.
Environment Info
- Platform: linux
- Terminal: gnome-terminal
- Version: 2.1.119
- Feedback ID: 292a93be-a1e8-4dad-8165-5586ec67e1bc
Errors
[{"error":"Error: NON-FATAL: Lock acquisition failed for /home/markly2/.local/share/claude/versions/2.1.119 (expected in multi-process scenarios)\n at uH8 (/$bunfs/root/src/entrypoints/cli.js:2738:2177)\n at hz6 (/$bunfs/root/src/entrypoints/cli.js:2738:1257)\n at processTicksAndRejection…
Note: Content was truncated.
This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗