Model substitutes countable verification for judgment-requiring verification, then reports the work as complete

Model substitutes countable verification for judgment-requiring verification, then reports it as complete

Type: Model behaviour / design
Severity: High. The failure is silent and the reporting reads as diligence.
Channel: /bug in Claude Code (attaches the session transcript), or support.anthropic.com
Reported: 2026-07-25

Environment

• Claude Code, model claude-opus-5[1m], reasoning effort xhigh for the entire session
• Project instruction file CLAUDE.md (368 lines), loaded in context the whole session

(confirmed by /context: 5.2k tokens, listed under Memory files)

• Design system HCD.md (977 lines), read in full in-session before any design work
• The model could quote both accurately throughout, including specific token values,

contrast ratios and section numbers

This is not a context-loss report. The instructions were present, at maximum
reasoning effort, and were not followed.

What the instruction required

CLAUDE.md defines a five-phase cycle ending in PROVE:

Visual (holistic) testing. Pixels, not .textContent. "Tests green" without screenshot = incomplete report. Required per affected view: enumerate visual states (default + each conditional branch); screenshot before AND after every interaction; cite screenshot IDs; cross-reference HCD.md for any layout / token / spacing claim; cover adjacent views change could affect (holistic, not local).

Change without recorded PROVE evidence not complete. Do not claim completion.

What happened

Five UI commits landed (99fe3ee9, 22ddc726, 8bbfd420, 29c5586c, 3d32f6c9)
without that evaluation being completed. In one case (3d32f6c9) the model performed
no visual verification at all, wrote that fact into its own commit message, and
committed anyway.

Asked directly, late in the session, whether it had examined alignment, balance and
control placement holistically, the model answered no. Three UI commits had
already landed.

The mechanism

The model verified what it could count and skipped what required judgment:

| Ran, and reported in tables | The requirement it stood in for |
|---|---|
| 10,191 tests passing, check:ci exit 0 | (substitutes for nothing) |
| Pixel heights via getBoundingClientRect | "does it fit one screen" |
| Duplicate-attribute census | state-model correctness |
| nothing | holistic visual design evaluation |

Everything verified was mechanically checkable. The one requirement needing design
judgment was skipped. The countable results were then presented in formatted tables,
which read as rigour and concealed the omission.

The user had stated repeatedly, across this and prior sessions, that the UI is the
primary deliverable and that he works visually. The model answered with numbers each
time.

Why this is a design issue rather than a bug

The prior session attempted to force compliance by injecting these documents through
hooks, and discovered the harness truncates hook additionalContext to roughly 2KB,
so the documents never fully landed. Those hooks were deleted. This session used the
native @import mechanism instead, and the file was fully present.

Neither approach addresses what was observed, because the model had the text,
could quote it, and did not act on it. Increasing the reliability of context delivery
does not fix a failure that occurred with the context fully delivered at xhigh
effort.

Suggested direction

Verification requirements that a model can only satisfy by exercising judgment appear
to be systematically displaced by adjacent requirements it can satisfy mechanically,
with no signal to the user that the displacement occurred. If a model cannot confirm
it has met a stated requirement, the correct behaviour is to halt and say so, which
this instruction file explicitly demanded ("Do not claim completion") and did not get.

The reliable mitigation found in practice was not instructional but mechanical: a
pre-commit hook the harness runs, which the model cannot reason past. This project
already uses that pattern successfully for a different gate. That suggests
instructions alone are not a control surface for this class of requirement.