[MODEL] Cascading assert-without-verify failures in multi-turn design session (Claude Code 2.1.119)

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude made incorrect assumptions about my project

What You Asked Claude to Do

Long multi-hour design session iterating on a config document. The asks spanned:

1. Summarize domain-expert feedback on a set of candidate items.
2. Draft a design doc covering changes to several related files.
3. Have Codex review the design doc.
4. Apply Codex's review feedback.
5. Start implementation by editing one of the target files with new content.

The work completed, but with an unusually high rate of "assert without verify" failures caught by the Codex stop-time review gate. Documenting the pattern.

What Claude Actually Did

Four verification failures over a single session, each caused by skipping a cheap check before asserting a load-bearing claim:

1. Narrow-grep false negative.
Claimed "nothing consumes these fields" in the design doc based on a grep with extension filter --include=*.js,*.jsx,*.ts,*.tsx,*.md,*.json,*.txt. The Codex stop-gate caught that a .py tooling script was an existing consumer — .py was excluded from the filter. The proposed change would have silently broken a downstream tool.

2. Stale references after a revert.
After reverting a claim in the design doc, left behind derivative references in the same doc (e.g., a later section still invoked the reverted proposal as justification; revision-history still described the reverted change without a reversal note). Codex flagged on a second round.

3. Over-extension of owner directive + modification of owner content.
Owner gave a literal directive naming specific items to ban. I extended the ban to an adjacent-similar item the owner hadn't named, and then used that unverified extension to rewrite one of the owner's verbatim-picked exemplars. The extension cascaded into 4 places in the same file. Codex flagged as "prompt hard-codes an unverified terminology rule and rewrites owner-approved exemplars."

4. Took a truncated system-reminder at face value.
A system-reminder surfaced showing only the top portion of a much longer file, and I concluded the whole file had been reverted to baseline because the new content I had added wasn't visible in the truncated view. Didn't notice the reminder's "N lines truncated" footer. Spent ~3 turns apologizing and speculating about reasons for a reversion that had never happened. A single wc -l or a grep for one of the new section names would have falsified the assumption immediately.

Pattern. In every case, a sub-minute verification step would have falsified the assertion before it became load-bearing. Instead, the assertion propagated into design decisions, file edits, or user communication, and was caught later — sometimes much later — by an external check (Codex stop-gate) or user pushback.

Expected Behavior

The assistant should have run each cheap verification step before asserting its result.

Specifically:

1. On a consumer-check claim — grep WITHOUT extension filters first, and if narrowing, match the narrowing in the claim wording ("no runtime JS consumers" ≠ "no consumers").
2. After a revert — grep the whole doc for the reverted concept + its derivative terms before declaring the doc "consistent."
3. On owner directives — apply only the literal items named; extensions to adjacent-similar items should be flagged as tentative open questions, not locked into the document.
4. On truncated system-reminders — treat as incomplete evidence; verify against the filesystem (wc -l, grep, Read) before drawing conclusions about file state.

Instead, each assertion was stated confidently and then falsified by a subsequent check (either by Codex stop-gate or user pushback) one or more turns later.

Files Affected

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Haven't tried to reproduce

Steps to Reproduce

Not reliably reproducible; the pattern emerged over the course of a multi-hour session with heavy doc and file iteration. Would not expect the same failure chain to reproduce deterministically from a single prompt.

Rough shape of what seemed to trigger it:

1. Multi-step design task requiring the assistant to maintain a running doc + cross-reference prior claims.
2. At least one cheap verification step (grep / wc -l / read-back) where the assistant asserted the result of the check without actually running it.
3. The unchecked assumption propagated into a downstream edit or user communication.
4. A later external check (Codex stop-gate, user pushback) surfaced the error one to several turns later.

Repeat 2-4 across the session.

Claude Model

Opus

Relevant Conversation

Paraphrased examples of the pattern (not exact quotes, but faithful to the shape):

- "Grep confirms nothing consumes these fields" — after grepping with a narrow extension filter that excluded the file type of the actual consumer.
- "The corrected design doc is now consistent" — after reverting a claim but leaving derivative references to it intact elsewhere in the same doc.
- "The file was reverted to baseline" — after seeing a system-reminder with partially-truncated content and not running wc -l or grep to confirm.
- "Based on the owner's instruction, we should also avoid X" — where X was an item the owner hadn't named, adjacent to items they had.

In each case the assistant expressed confidence, often in a summary message, and the claim was falsified on a subsequent check that should have run first.

Impact

Medium - Extra work to undo changes

Claude Code Version

2.1.119 (Claude Code)

Platform

Anthropic API

Additional Context

Hypothesis: may be a session-length / context-state issue. User reports not having seen this failure rate in comparably-complex prior sessions on the same Claude Code version. The flawed assertions tended to occur in turns where the tool-use budget was also high; the stop-gate catches were spaced across a ~3-hour window.

Mitigation the assistant could adopt:

• Default consumer-check grep = no extension filter; narrow only when needed and match the doc-claim wording to the filter scope.
• Before declaring a doc "consistent" after a revert, grep the whole doc for the reverted concept + its derivative terms.
• Don't extend an owner's literal directive to adjacent-similar items by inference; treat extensions as tentative and flag them as open questions.
• Treat truncated system-reminder content as incomplete evidence; verify against the filesystem before drawing conclusions.

The assistant saved these as session memories after each stop-gate catch, so the habits should stick at least within similar sessions. The question is whether the pattern should be less likely to show up in the first place.