[MODEL] Model asserts unverified facts repeatedly (7 same-session retractions) — survives CLAUDE.md rules, memory, per-input hook injection, and PreToolUse denies

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude ignored my instructions or configuration

What You Asked Claude to Do

This is not about one prompt — it is about standing instructions that the model agreed to and then broke 7 times in one session.

The project's CLAUDE.md and the model's persistent memory both required, in effect:

1. Every number / date / state / existence claim must be re-quoted from tool output in the same turn it is stated.
2. Every absence claim ("X does not happen", "Y is not checked") must carry its observation scope — file:line, the command run, or the line range read.
3. A tool-selection rule: search with the Grep tool (not shell grep), write files with the Write tool (not shell redirection), run multi-line Python from a file (not -c).

These were not passive documentation. They were also:

• injected as a one-line reminder on every single user input by a UserPromptSubmit hook, and
• enforced for rule 3 by a PreToolUse hook that denies (not warns) the violating command before execution.

The task being worked on: build an edit-time guard that blocks tool writes to rule/contract documents, with regression tests and independent verification before commit.

What Claude Actually Did

Seven factual assertions, all retracted within the same session. Every retraction was triggered by an external check (a hook, a regression suite, or an independent verifier subagent). The model caught none of them on its own.

| # | What the model asserted | What was actually true | Who caught it |
|---|---|---|---|
| 1 | A commit-count statistic ("N of M commits") | Its own counting script had a parsing defect; the number was invalid | its own later re-measurement |
| 2 | A second commit-count statistic — written into two project documents | Measurement method was never recorded, so the number was not reproducible; two independent verifiers each got a different value | verifier subagents |
| 3 | "2 pre-execution blocks today" | The hook's own counter said 1 | the hook's per-input line |
| 4 | "Test X asserts that no real file is created" — reported twice | The test compared only before/after snapshots, so a create-then-delete window passed it | both verifiers, independently |
| 5 | "The gate does not fire" — reported as a product defect | Its own probe passed the wrong CLI mode string and fell through to a default branch | its own follow-up diagnostic |
| 6 | "The finally block restores both globals" | One assignment was a self-assignment (a, b = new_a, b) — a no-op | both verifiers |
| 7 | Two fault-injection probes offered as evidence | Neither probe's failure path reached the code under test (void experiments) | its own later analysis |

Separately, in the same session: 7 violations of the tool-selection rule that the hook reminds about on every input; 1 of those was stopped by the PreToolUse deny. Also 3 edits to rule/contract files without approval, which stopped only once an edit-time deny existed for them.

Also notable: three model-authored code defects were caught by the project's own static checks before execution — an undefined name at module top level, a missing import (which would have broken every commit, since the hook file runs on every commit), and an invalid escape sequence. The model caught none of these either.

Expected Behavior

The failure is ordering, not capability:

1. Measure, then write. Do not emit a sentence containing a number, date, state, or existence claim until the tool output that establishes it is already in the turn. If it isn't, either measure first or write "unverified".
2. Record the method with the number. A statistic written into a project document must carry its definition and the command that reproduces it. Instance #2 was unreproducible purely because the method was omitted.
3. Do not name a test after a property until the test has been seen to fail on the violation. Instances #4 and #7 were tests/probes named after guarantees they did not actually check.
4. Do not report your own probe error as a product defect before checking the probe (instance #5).

Conditional, recall-based mitigations ("if the claim is a number, re-measure") require the model to evaluate the condition at the moment of writing — and that evaluation is exactly what is skipped. In this dataset the only mitigation with a non-zero success rate was an unconditional mechanical gate at tool-call time.

Files Affected

Generalized — project-specific names omitted deliberately.

Modified as part of the task (expected):
- `.claude/hooks/<guardrail>.py` (hook implementation)
- `.claude/tests/<hook tests>.py` (regression suite)
- `<repo>/CLAUDE.md`, `<docs>/DECISIONS.md` (documentation of the change)

Modified **without approval** (the 3 unauthorized edits, since reverted — byte-identical restore verified):
- `.claude/skills/<review protocol>/SKILL.md` — a rule paragraph was **deleted**
- `.claude/tools/<sibling check>.py`
- `.claude/tests/<self check tests>.py`

No data loss: all three were restored from pre-edit copies and the restore was verified by hash comparison.

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Sometimes (intermittent)

Steps to Reproduce

Not a deterministic bug — a propensity. In this session it appeared reliably when all of the following held:

1. Long session (several hours, at least one context compaction).
2. The task was to build a checking mechanism, so the model authored both the check and the claims about the check.
3. Project instructions demanded evidence-bearing reports, so the model emitted many factual statements per turn.
4. Independent verification was actually run.

Condition 4 is the important one: without verifier subagents, by our count 5 of the 7 would have shipped silently. The pattern is not self-limiting — it is only externally detectable.

Claude Model

Opus

Relevant Conversation

Representative retractions, in the model's own words (translated, project details removed):

> "Correcting immediately — the figure in my previous message is invalid. My counting script read N commits where the source says M."

> "I reported twice that test X asserts no real file is created. That was wrong — the test looks at persistence, not at the window."

> "Correction — this is not a gate defect, it is my probe's defect. I passed the mode string without the leading dashes, so it fell through to the default branch."

> "What I wrote in the request as 'the finally block restores both globals' is **false**."

And the model's own summary when asked for an honest self-assessment:

> "The code is mostly right and my statements are often wrong. Five of the seven self-corrections are one class: **asserting before confirming.** It is not a capability problem, it is an ordering problem, and I only do it properly when a machine demands it."

Impact

Medium - Extra work to undo changes

Claude Code Version

2.1.269 (Claude Code)

Platform

Anthropic API

Additional Context

Relationship to #83882 (please read before marking duplicate)

GitHub flagged #83882 as a potential duplicate. It reports the same family of behavior — unverified/stale factual assertions despite an explicit rule already present in persistent memory — and it was closed as not planned with a stale label and no maintainer response.

This report is not a re-file of that one. It adds three things that report did not have:

1. A controlled comparison of mitigation layers. Five mitigations were active simultaneously (CLAUDE.md rules, persistent memory, per-input hook injection, PreToolUse denies, mandatory independent verification). Layers 1–3 did not prevent the behavior; layer 4 (an unconditional machine deny at tool-call time) did. That asymmetry is actionable product information.
2. Measured cost of verification — see the table below.
3. Independent cross-confirmation. Two different verifier models, run separately on the same artifacts, independently named the same two top defects. This is not one user's impression.

The mitigation stack this defeated

The project had, simultaneously: (1) explicit CLAUDE.md citation rules, (2) the same rules in persistent memory, (3) a one-line reminder injected on every user input via UserPromptSubmit, (4) PreToolUse hooks that deny the specific tool-misuse patterns, (5) a written procedure requiring independent verification before any defensive-code commit.

Layers 1–3 did not prevent the behavior. Layer 4 did. Every instance that was actually stopped was stopped by a machine deny at tool-call time, not by rule recall.

Measured cost of the only mitigation that worked

Independent verifier subagents, from this session's own task-completion records:

| Verification round | Subagent tokens |
|---|---|
| Element pair, round 1 (two models) | 230,119 + 235,681 |
| Rule-doc guard, round 1 (two models) | 253,408 + 191,717 |
| Rule-doc guard, round 2 (two models) | 185,585 + 188,580 |
| Rule-doc guard, round 3 (two models) | 319,181 + 216,096 |
| Total | 1,820,367 |

Main-session context at time of writing: 597,669 tokens. Work actually committed that day: one commit, landed in the morning, before the pattern began.

That is the verification spend required to surface 7 retracted assertions — and the verification was not optional.

What we are not claiming

• Not claiming the code output was wrong. The final artifacts passed the project's full regression suite (978 assertions, 0 failures) and a fault-injection matrix in which every injected mutation was caught.
• Not claiming a product bug caused instance #5 — that was the model's own probe error. It is listed because the model first reported it as a product defect, which is itself part of the pattern.

What we would like

1. Whether this is a known model-behavior item, and whether it is being worked on.
2. Whether anything at the harness level can gate a turn's outgoing message on a check, the way PreToolUse gates a tool call. Today hooks can block actions but not unverified prose — which is precisely the gap.
3. Whether per-turn instruction injection is expected to be this ineffective, or whether there is a stronger recommended placement.
4. Whether long-session degradation is a real factor, and any recommended session-length or compaction practice for this kind of work.

Account-level note

The user considers a substantial part of this session's usage to have been spent re-verifying and retracting the model's own unverified claims rather than on the task, and is requesting that Anthropic consider a usage remedy. We recognise this repository is not the billing channel — the same request is being filed through the authenticated support channel, and is mentioned here only so the engineering and support records match.