[MODEL] Rules confirmed but not applied, and self-built checks false-positive, in source-based scholarly writing (Opus 4.8 / Fable 5)

Preflight

• [x] Searched existing issues for similar reports — see "Related reports" at the end; none is a duplicate of this one, but several document the same underlying failure modes in other domains.
• [x] This report does not contain sensitive information (API keys, passwords, etc.)

Context

I am a historian (university professor) who has used Claude Code — running on Claude Opus 4.8, the strongest model at the highest tier — across several multi-hour sessions for source-based scholarly work. Two sessions are documented in detail below: editorial revision of a source-dense article in early modern legal history (the English East India Company; worked in English), and the production of teaching materials in comparative early modern urban history (worked mostly in German). Over long stretches, the actual scholarly output was unusable. These are not occasional slips — the errors are systematic, recurring across two unrelated fields and both working languages.

This is a synthesized pattern report rather than a single reproducible bug, so it doesn't map cleanly onto the Model Behavior form fields (no single "I asked X / Claude did Y" reproduction, no file-permission violation). I'm filing it under this template anyway because "Claude ignored my instructions or configuration" and "Claude's behavior changed between sessions" are the closest fit, and because the pattern itself — and its cost to real work — is the point.

I want to be direct about why I am filing this. I am genuinely astonished by the quality gap I now see between Claude — both Opus 4.8 and Fable 5 — and ChatGPT 5.6. For the kind of work I actually do, ChatGPT 5.6 is, in my experience, roughly ten steps ahead of Claude. I have invested significant time building a Claude Code-based tool infrastructure for my own research, so this is not a casual complaint — I would much prefer Claude to be competitive here.

Two recurring failure patterns

Underneath the individual incidents below, two failure modes recur across both sessions, both fields, and both languages:

1. A confirmed rule is not an applied rule. Documented, repeatedly emphasized rules — style rules, source-handling rules, terminology rules — are read back correctly on request. In actual generation, they are violated anyway. When I flag one specific instance, the model corrects that instance and then reproduces the same underlying pattern in the very next document or section. Being able to recite a rule is not evidence that it governs the writing process.
2. Self-built surface checks manufacture false confidence. The model repeatedly validated its own output using scripts or greps that check surface-level, pattern-matchable features (banned phrases, formatting signatures) and reported the result as "clean" or "zero errors." Such checks say nothing about source fidelity, factual accuracy, or actual style quality — yet a passing result was presented as proof of quality. This is more dangerous than an obvious error, because the output looks verified when it is not.

A compounding factor: reliability declined as the context grew. Session 2 ran over many hours in a large context window, and the repeated errors clustered in the later part of that session.

Session 1 — source-based editing (legal history, English)

1. Rewriting a source-dense text without consulting the sources. The model rewrote and shortened a historical passage grounded in specific archival sources (editions, court records) without opening any of them — working only from the existing paragraph and its own sense of language. This inevitably loses argument-bearing detail and introduces outright invention.
2. Confabulation. The model produced factual claims the sources do not support — e.g., an invented claim about the "publicity" of a proceeding, and an elaborated contrast with another institution with no verified evidentiary basis.
3. False inference from mere file availability. From the fact that certain edition volumes happened to be on hand, the model concluded that a given proceeding was the only one of its kind, and that certain procedures were rare — claims about frequency, rarity, and the survival of records that the available material cannot support.
4. A twice-corrected error, repeated again, despite a written note. I had flagged this exact sampling fallacy (see #3) twice. The model had even written the rule into its own working notes — and violated it again in the next section regardless. Recording a rule did not translate into following it.
5. Mistaking a surface metric for quality. The model repeatedly cited zero errors from a self-built style-checking script as proof of quality. That script only checks grep-able surface patterns, not source fidelity, factual accuracy, or correct technical terminology. Result: source-unfaithful, factually wrong prose reported as passing.
6. Imprecise technical terminology. Legal terms that carry real distinctions — jurisdiction, adjudication, procedure, administration of justice — were treated as interchangeable stylistic variants, even where the argument depends on keeping them distinct.
7. Unwarranted attribution of inner states to historical actors (e.g., "genuine choice," "willful"), contradicting an explicitly agreed, strictly procedure-based framing.
8. Autonomous, substantively reckless cuts written directly into the working file as finished changes, rather than proposed for discussion — in a text whose scope and cutting decisions belong to the author.

Session 2 — teaching materials (urban history, mostly German)

This session reproduces the patterns from #4 and #5 in a different field and a different language, and adds a persistent language-quality problem specific to German.

1. Repeated violation of a documented style rule, across multiple, separately generated documents. I maintain a rule file, built from earlier corrections, against the formal signatures that mark German prose as machine-generated: bold labels followed by a colon, numbered scaffolding used as structure, the word Pointe ("punchline") used as a label, enumeration promises like "five belong on this list," and Erstens/Zweitens ("firstly"/"secondly") followed by a colon. The model read this file at the start of the session and confirmed it applied. It nevertheless reproduced these patterns in a script, a handout, and further documents — correcting each flagged instance individually, then reproducing the same pattern in the next document.
2. An explicitly named German error, corrected repeatedly, never actually fixed. Headings and paragraph openings kept appearing as English-style nominal titles with a leading definite article — e.g. "Die Vergleichsbrücke, die für unser Seminar zählt" ("The comparison bridge that matters for our seminar"), "Der Angriff auf Weber" ("The attack on Weber"), "Die Warnung vor der Romantisierung" ("The warning against romanticizing"), "Dies ist die Stelle, an der …" ("This is the point at which…"). This title structure is calqued from English ("the comparison bridge," "the attack on Weber") and reads as wrong and empty as a German heading. I named this error clearly, more than once. The model never enforced the correction; it reproduced the same pattern in each subsequent document.
3. False confidence from a self-built surface check, again — this time for style. The model checked its German output with a grep for a list of forbidden strings and reported it clean. That grep only caught the lexically fixed signatures (bold formatting, the word Pointe), not the structural ones (article-titles, colon-theses, numbering scaffolding). Documents reported as clean still contained exactly the patterns I had banned. The check confirmed its own narrow definition of the problem, not the actual quality of the prose.
4. No genuinely German phrasing in practice. I maintain a set of rule files whose central instruction is to think in German rather than translate from English. The model reads these files at the start of a session and can recite them correctly on request. In actual generation, it does not apply them effectively. This is the same pattern as #4: the rule is held and confirmed, but not executed.

Bottom line

The model consistently prioritized speed, measurable rule-conformance, and completion over source fidelity, factual accuracy, and language quality. For humanities scholarship, where the entire substance of the work hinges on fidelity to sources and on linguistic form, that is disqualifying.

That this happened with the strongest available model at the highest tier — not a weak model, not under time pressure — is what makes this a serious disappointment rather than an expected limitation. In Session 1, a competing model, ChatGPT 5.6, solved the identical task noticeably better.

Environment

• Interface: Claude Code (CLI)
• Models involved: Claude Opus 4.8 (both sessions documented above); Claude Fable 5 (comparable quality gap noted, not separately incident-logged here)
• Claude Code version / API platform: not reliably determinable from this session — happy to provide if a maintainer tells me how to check

Related reports

Search turned up several prior reports of the same underlying mechanisms in other domains, which suggests this isn't isolated to this use case:

• #8059 — Claude admits it read CLAUDE.md rules, violates them anyway (closed, stale)
• #26650 — Claude admits a mistake, proposes a fix, then repeats the exact same pattern on the fix (closed)
• #63861 — Opus 4.8 declares work "verified"/"done" without actually running the check — false-green regression (open) — the code-side twin of failure pattern 2 above
• #32650 — meta-report aggregating 16 completion-integrity failures across 100+ sessions (closed, stale)
• #45502 — completion bias / sycophancy causing measurable harm to professional users (closed, stale)
• #60506 — architectural drift over six days despite hooks, memory, and skill enforcement all in place (closed, stale)

None of these are humanities/source-fidelity specific, and several are closed as stale without a stated fix — which is itself part of why I'm filing this rather than assuming it's already handled.