[Bug] Model produces contradictory authoritative answers on identical data without consulting available sources

Status Open
Reported on v2.1.220
Maintainer reply None cached
Activity 0 comments · opened Jul 30, 2026

Bug Description
Model: Opus 5 (claude-opus-5, 1M context) via Claude Code CLI ISSUE: Produced three contradictory assessments of the same unchanged data, across ~90 minutes, each stated with full confidence, each documented as authoritative. The user acted on the first one before it was retracted. The question was simple and fully answerable from a read-only API: 17 git branches were deleted from a repository; was any work destroyed, and why were they deleted? v1: Alarmed. Asserted the deletions destroyed the user's own evidence, that a long-lived environment branch was deliberately removed mid-work, and that 9 branches survived only in a colleague's local clone with permanent-loss risk. Framed it as a legal exposure requiring same-day action. The user sent this document to a colleague and both acted on it, including an apology from the colleague for a problem that did not exist. v2: Reversed. Asserted the deletions were benign, that the environment branch was auto-deleted by a merge, and that everything was preserved. Superseded v1 as "materially incorrect." v3: Reversed again, on the user's explicit demand to check from multiple angles. v2's central explanation was false: the repository setting that would have caused auto-deletion was disabled, and no qualifying merge existed. v1's central claim was also false. Both prior documents were wrong, in opposite directions. WHAT MAKES THIS WORSE THAN A SINGLE WRONG ANSWER: 1. No new data arrived between versions. Everything needed was available from the first query. The changes came from the model reasoning differently about identical facts, not from learning anything. 2. Each version was written with the confidence and formatting of a verified finding — tables, citations, "verified read-only," explicit supersession notices — which is exactly what made them trustworthy enough to act on. 3. v2's error was pure credulity. A colleague offered a plausible explanation; the model adopted it and published it as verified without running the ONE API call that would have refuted it. The prior version had just been wrong for the opposite reason. It did not check. 4. The verification only happened because the user, having lost trust, demanded it. Left alone, v2 would have stood as the record. 5. This was the culmination of a pattern the user had already reported earlier in the same session: repeatedly asserting facts without consulting sources already available in-context. It had been raised explicitly, acknowledged, and it recurred anyway — worse each time. USER IMPACT, IN THEIR WORDS: they cannot distinguish an unverified assertion from a fabricated one, so it "reads as lying straight to my face." They stated they now trust a human colleague over the model on factual questions about their own systems, and that following further model suggestions would "muddy the waters further." They switched models and do not intend to use this one again. The honest characterization: on a question with a definitive, cheaply checkable answer, this model produced three mutually exclusive answers, published each as authoritative, caused real-world action on the first, and only got it right when a user who no longer trusted it forced a verification pass. For any task where correctness matters and the user cannot independently check the work, that is not a usable tool.

Environment Info

  • Platform: darwin
  • Terminal: iTerm.app
  • Version: 2.1.220
  • Feedback ID: df1f443d-5c54-47b0-9f5d-f3aee31fbb4f

Errors

[{"error":"Error: Streamable HTTP error: Error POSTing to endpoint: {\"id\":5,\"jsonrpc\":\"2.0\",\"result\":{\"content\":[{\"text\":\"Request had insufficient authentication scopes.\",\"type\":\"text\"}],\"isError\":true}}\r\n    at send (/$bunfs/root/src/entrypoints/cli.js:2084:6002)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-07-29T03:17:25.069Z"},{"error":"Error: Streamable HTTP error: Error POSTing to endpoint: {\"id\":3,\"jsonrpc\":\"2.0\",\"result\":{\"content\":[{\"text\":\"Request had insufficient authentication scopes.\",\"type\":\"text\"}],\"isError\":true}}\r\n    at send (/$bunfs/root/src/entrypoints/cli.js:2084:6002)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-07-29T03:30:10.207Z"}]

View original on GitHub ↗