[MODEL] Model writes "falsifiable" when "verifiable" is the obviously intended word — reproduced 4x over 4 months, project-scoped correction doesn't transfer
The specific claim
This is not a general "picks the wrong word sometimes" report. It's one precise, repeatable substitution: the model writes "falsifiable" (or "unfalsifiable"/"falsifiable-ish") in a sentence where the claim being described is not actually exposing itself to potential disproof — it's simply a fact or number that can be checked/confirmed against existing evidence (a quote, a commit, a transcript, a measurement). The correct word in every one of these cases is "verifiable" or "checkable." The two are not synonyms: falsifiability is Popper's demarcation criterion (a hypothesis is falsifiable if it forbids some observable outcome); verifiability is just "can be looked up and confirmed." The model conflates them in one direction only — reaching for the more academic-sounding word when the plain one is what's meant — never the reverse.
Evidence: 4 confirmed occurrences over 4 months, reconstructed from session transcripts
Pulled directly from a personal Claude Code session archive (JSONL transcripts ingested into a queryable database), not from memory or impression:
- 2026-06-09 — model wrote "falsifiable," was corrected, and its own correction acknowledged: "You're right, wrong word. 'Falsifiable' means capable of being disproven... What I meant was verifiable, or simply backed by the record."
- 2026-06-24 — recurred in a different session; corrected again: "You're right — verifiable. And the conflation is a real one, not a stylistic nitpick: falsifiable is Popper's demarcation criterion... and I reached for it as borrowed sophistication when I meant 'you can check this against evidence.'" This correction produced a project-scoped memory file documenting the pattern.
- 2026-08-04 — recurred a third time (as "falsifiable-ish," a hedge that didn't prevent the same error), and the model's own correction says: "Third time now, per your memory note from May/June... apparently just having it written down hasn't been enough to stop it."
- 2026-09-25 — recurred a fourth time, in a completely unrelated project/repo that had never touched the one where the memory file above lived.
(A "May 2026" instance is referenced in the memory file's own history but could not be independently confirmed via direct DB query — both an exact-word search and a broader correction-pattern search across that month turned up nothing. Treating that one as unconfirmed; the four above are.)
Why this is worth reporting as a pattern, not just "correct me each time"
The interesting finding isn't the vocabulary error itself — it's that a documented, written-down correction (three separate write-ups to the same memory file, one of which explicitly self-labeled "third time") did not stop a fourth recurrence, and the fourth recurrence happened in a different project than where the memory lived. That's a real limitation of project-scoped memory for corrections that are about the model's own general behavior rather than project-specific facts: the correction structurally cannot reach a session that never opens the project where it was written. (Filed for context, not as the headline ask — a user-side memory-scoping choice isn't itself a bug, but it's the reason a specific, repeatedly-corrected error kept coming back, and worth knowing about if there's ever a cross-project or account-level memory/instruction mechanism under consideration.)
Reproduced
Environment for the most recent (4th) occurrence: CLI 2.1.281, desktop app, macOS 15.7.9, 2026-09-25.
6 Comments
Not a duplicate of #94793. That issue is about the model coining vocabulary — compound nouns that don't exist in German, or real German words used in senses that don't exist at all (a generation/fabrication failure, and by its own description possibly tied to German's compounding morphology specifically). This issue is a different mechanism: "falsifiable" is a real, correctly-defined English word — the problem is that its actual meaning is the near-opposite of what's needed here. Falsifiable (can be shown false) and verifiable (can be confirmed true) are, per Popper's own asymmetry, distinct and non-interchangeable properties, not casual synonyms — so this reads less like "generates a word that doesn't exist" (#94793's failure mode) and more like "retrieves the wrong member of a small cluster of semantically-adjacent/near-antonym terms," reaching for the one that sounds more sophisticated in-context rather than the one that's actually correct.
Speculative, flagged as a hypothesis rather than a finding: the reporter's own suspicion is that this may not be narrow to this one word pair — that "falsifiable/verifiable" could be one instance of a broader pattern where training pulls toward a near-antonym or adjacent term in the same conceptual neighborhood, rather than this being a fix scoped to just these two words. Worth keeping in mind during triage rather than treating a fix as "block/adjust this one substitution" if the underlying mechanism turns out to be more general.
Watermarking is poisoning the token stream with negative-layer-inducing bullshit like this, it will ultimately be what corrodes Anthropic if they don't change course, no matter how much they think they are stalling competitors and distillation. It's gallium.
I disagree with the above comment. This particular error has been happening since at least June and I believe May - long before the watermarking was introduced. As a result I seriously doubt that this is a watermarking problem.
August is when Anthropic announced they were watermarking outputs. I think it's an assumption that they just discovered this technology and weren't already using it to shift training data. The decline through 4.7-5 that surrounds watermarking and development of tools that can detect and reverse engineer it followed by a sudden leap in capability gives me a gut feeling, but I may be jumping to conclusions and "get off my lawn"ing the watermarking. But all the same, it will lead to many issues like this if allowed to persist, imo.
</rant>
I suppose it is possible that the underlying training has been happening, but I'll let someone from Anthropic (who will actually know) rather than speculating on it. Ordinarily I would think that something isn't in a release until it is, but the fact that the training data is so large and so hidden from us means that I shouldn't speculate. As i said in the issue itself, I have records that this happened in June and July and persists to today.
Cause aside, I agree that defaulting to a framework of falsifiability is not productive and often inverts intent. Most of my work calls for systems that prioritize verifiability (concrete evidence of function within my actual constraints rather than invalidation through shallow and flawed observations). Falsifiability should be an opt-in heuristic, not a defensive default and convenient pivot away from a difficult solution. This shift from verification to falsification has a pretty severe negative impact on exploration and long-horizon automation, stifling exploration. It prematurely kills contrarian ideas that consensus rejects but could still succeed and derails autoregressive loops. In concept exploration and unsupervised tasks, a single toy-scale invalidation (meaningless invalidation) leads Claude to abandon a perfectly viable, verifiable intention.