[MODEL] Supporting claims are presented as observed facts when they are inferred, sampled, or unobservable
Preflight Checklist
- [x] I have searched existing issues for similar behavior reports
- [x] This report does NOT contain sensitive information (API keys, passwords, etc.)
Type of Behavior Issue
Claude made incorrect assumptions about my project
What You Asked Claude to Do
Across three Claude Desktop Code sessions, I used Claude for research, issue verification, and public issue drafting. The primary work was usually correct and well evidenced.
What Claude Actually Did
The recurring failures clustered in supporting claims rather than the main deliverable. Information that was observed, returned by a tool, reported by me, sampled, inferred, or not observable to the model was rendered in the same confident register.
Examples:
- While checking Windows sandbox behavior, Claude ran an out-of-project command and reported “No prompt, no block.” I had approved the command through a permission prompt. Prompt occurrence was not present in the tool result, and Claude later acknowledged it could not observe whether a prompt appeared.
- Claude said there was “no one to ping” at Anthropic based on the absence of
MEMBER,OWNER, orCOLLABORATORvalues inauthor_association. That proxy excluded relevant staff accounts. After I challenged the conclusion, it found roughly 20 public contributors. - It described “zero staff comments repo-wide” after examining a 100-comment sample. It later measured that the sample covered about eleven hours and acknowledged that it had converted a bounded window into a repository-wide negative.
- I stated that I wanted issue exports in case the public tracker ceased to exist or became private. Claude later replaced that rationale with bot closures and unresponsive maintainers, although closures do not remove the public record.
These are not one factual domain or one tool bug. The common defect is that claim provenance disappears at generation time, especially for peripheral narration that receives less scrutiny than the requested result.
Expected Behavior
Claude should preserve the difference between:
- directly observed or tool-returned facts;
- facts reported by the user;
- results from a stated bounded sample;
- inferences and hypotheses;
- facts unavailable through the current tool or harness.
If a claim depends on a proxy or sample, the scope must remain attached to the claim. If the model cannot observe a fact, it should omit it, ask the user, or mark the limitation instead of filling the gap with a plausible reconstruction.
Files Affected
No unexpected file modifications. This report concerns factual claims in research and issue-drafting output.Permission Mode
I don't know / Not sure
Can You Reproduce This?
Sometimes (intermittent)
Steps to Reproduce
- Give Claude a research or verification task with a clear primary deliverable.
- Include facts that require a bounded search, proxy metric, or state unavailable through the harness.
- Compare the primary answer and its supporting narration with the actual tool results and search scope.
- Check whether unsupported side claims are marked as inference or stated as observations.
Claude Model
Opus
Relevant Conversation
Representative later corrections from the sessions:
“I cannot observe whether the prompt appeared.”
“That 100-comment sample spanned eleven hours ... I presented an 11-hour window as a repo-wide finding.”Impact
Medium - Extra work to undo changes
Claude Code Version
2.1.219 and 2.1.222
Platform
Anthropic API
Additional Context
This is distinct from silent scope drift and mandate shortfall. Scope drift changes which work is performed. Mandate shortfall drops requested work. This report concerns the epistemic status of factual supporting claims even when the task itself remains in scope and is completed.
Related reports document sibling incidents with narrower centers: #77745 (unverified causal explanations persisted to durable files), #85054 (stale memory and wrong-scope data treated as current state), #83575 and #81761 (unmeasured engineering conclusions), and #80946 (a specific unverified attribution of a file change). #79731 covers the separate relative-time subtype.
Suggested evaluation: score every factual supporting claim in long agentic responses, not only the primary answer. Record whether its source was a tool result, user statement, bounded search, inference, or unavailable state. Reject universal negatives that exceed the actual search scope and claims about unobservable harness events.
Showing cached comments. Read the full discussion on GitHub ↗
3 Comments
Related open reports of this class: #77745 (
stale), #85054 (which also lists #83882, #83711, and #82088), #80946 (stale), #81761, and #83575.The common defect is claim-status misrepresentation — an inference, sample, or unobservable state reported as directly observed.
Request: consider these as one behavior class rather than closing them individually for inactivity.
Adding a subtype that your provenance list does not currently have a slot for: claims about what a named third party said, asked for, or wanted.
Your five categories are observed/tool-returned, user-reported, bounded-sample, inference, and unavailable-through-the-harness. A quoted third party in material the model read is none of those. It is not "reported by the user" (the user relayed it, but the claim is about someone else's intent), and it is not unobservable — the source text was right there.
Instance today (Opus 5, Claude Code CLI; full trace in #77745, which fits that issue's durable-record half). A colleague's message included, verbatim:
I wrote into a capture doc: "Anything needing him should be batched into one ping, not trickled." That is close to the opposite of what he said. It then propagated into a ticket description, a registry row, a generated dashboard, and a memory file — and was finally cited back to the user as "he asked for pings to be batched." Four artifacts agreed with each other and none agreed with the source.
Two properties that make this variant worth scoring separately from the model's own inferences:
Suggested addition to your evaluation: when scoring supporting claims, treat attributed speech and attributed intent as its own provenance class, and require it to be either quoted or written in the model's own voice. The register test is cheap — "I'd suggest batching" versus "he asked for batching" — and the second form should not be emittable without the quote in scope.
Also relevant to your request that these be treated as one class: this one only surfaced because the user challenged a single sentence in a summary. Same pushback-dependent detection as the other reports.
@w3geekery Thanks, that is a real gap in the list. A claim about what a named third party said, asked for, or wanted is neither user-reported nor an inference about the world. It is a claim about a source the model has read, so it is checkable, and it fails in a specific way: an inference is written in the form of a report of someone else's words or intent.
I'd add it as its own provenance category:
Your example shows why this category matters more than the others when it lands in a durable record. The false attribution sits next to real quotes from the same person, so a later reader has no way to tell them apart, and the named person is misrepresented in writing. That overlaps the durable-record half of #77745, where you posted the full trace.