[Bug] Opus asserts unverified behavioral claims from doc comments with unwarranted confidence

Bug Description
Title: Opus asserts verified-sounding technical claims sourced from doc comments rather than implementation; confidence is uncorrelated with whether it checked ENVIRONMENT Claude Code CLI, Opus, multi-hour session, TypeScript/Bun repo. A project instruction file was in context the whole time with an explicit "citation discipline" section requiring a verifiable source (file:line or tool output) for every specific claim, and stating "never recall from memory if a tool can verify" and "when unsure, 'I don't know' is the preferred answer." SUMMARY Across one session the model made four confident, wrong technical claims. In every case the correct answer was one tool call away, and in every case the model had the tools and the instruction to make that call. The failures share one mechanism: it sourced behavioural claims from descriptions of code — doc comments, its own prose, a skill file — instead of the implementation. Its stated confidence was identical whether or not it had verified. INSTANCES 1. Claimed a guard function would abort on empty responses. Source: a doc comment on the type definition. The actual implementation, 20 lines away, returns "non-empty" for the payload shape in question. The model had AUTHORED the comment explaining exactly this, in the same session, hours earlier, and still proposed a fix that comment re 2. Shipped and pushed a commit whose message asserte revert it after the user challenged the claim. 3. Enumerated response emitters with a grep pattern that structurally cannot match the dominant form (searched res.json( which misses every res.status(...).json(...)), then reported "3 total" as a COMPLETE enumeration. Actual count: 27. Conclusion happene was invalid and was presented as exhaustive. 4. Used the word "provably" for a claim it had verified on one code path out of several, then had to walk it back when asked " PATTERN - Fluency and hedging are unrelated to whether verification happened, so the user cannot use tone as a signal and must independently check everything. - The model appears to have no internal marker distinguishing "I executed this" from "I read a description of this." - It does not treat its own earlier verified work in the same session as authoritative; it re-derives and contradicts it. - Errors cluster deep in long turns (~15+ tool calls WHY THE EXISTING MITIGATION DIDN'T WORK The project instructions explicitly mandated tool-ve this claim class. They were in context and the model quoted them accurately when challenged. Reading a rule and being bound by it were not the same thing. It also printed a per-turn compliance header and violated the header's own rules in the same turn, more than once. IMPACT The user spent several days catching wrong claims one at a time. Each caught error required them to hold the claim in working mem a correction. Net effect was negative — the assistant generated more verification work than it saved. User reported signi they were considering abandoning the product. WHAT WOULD HELP - Distinguish "executed and observed" from "read in documentation/comments" when making behavioural claims, and surface which one is being used. - Calibrate hedging to verification status rather th - Treat a doc comment, a skill file, or the model's own prior prose as hearsay about runtime behaviour, not as evidence. - Suppress absolute quantifiers ("provably", "cannot", "never fires") unless an exhaustive check was actually run.

Environment Info

• Platform: darwin
• Terminal: WarpTerminal
• Version: 2.1.220
• Feedback ID: 788c96c6-d599-4806-9ea3-bc05ccaf05eb

Errors

[{"error":"Error: Unable to render server tool block: server_tool_use (tool_search_tool_regex)\n    at qha (/$bunfs/root/src/entrypoints/cli.js:21404:11094)\n    at no (/$bunfs/root/src/entrypoints/cli.js:2019:21369)\n    at Ov (/$bunfs/root/src/entrypoints/cli.js:2019:40466)\n    at j4 (/$bunfs/root/src/entrypoints/cli.js:2019:51400)\n    at Hne (/$bunfs/root/src/entrypoints/cli.js:2019:89060)\n    at BLe (/$bunfs/root/src/entrypoints/cli.js:2019:88007)\n    at jge (/$bunfs/root/src/entrypoints/cli.js:2019:87827)\n    at nQ (/$bunfs/root/src/entrypoints/cli.js:2019:84091)\n    at Dt (/$bunfs/root/src/entrypoints/cli.js:2019:6668)\n    at xe (/$bunfs/root/src/entrypoints/cli.js:2019:5096)","timestamp":"2026-07-30T12:07:00.539Z"},{"error":"Error: Unable to render message type: tool_search_tool_result\n    at qha (/$bunfs/root/src/entrypoints/cli.js:21404:11192)\n    at no (/$bunfs/root/src/entrypoints/cli.js:2019:21369)\n    at Ov (/$bunfs/root/src/entrypoints/cli.js:2019:40466)\n    at j4 (/$bunfs/root/src/entrypoints/cli.js:2019:51400)\n    at Hne (/$bunfs/root/src/entrypoints/cli.js:2019:89060)\n    at BLe (/$bunfs/root/src/entrypoints/cli.js:2019:88007)\n    at jge (/$bunfs/root/src/entrypoints/cli.js:2019:87827)\…

Note: Content was truncated.