Correct post-mortem, same-class recurrence: model names a verification-gap failure, 'fixes' it by grading a different proxy, and reproduces the identical class on the next step

Open 💬 0 comments Opened Jul 11, 2026 by Mig-Sornrakrit

The model diagnoses a failure class correctly, "fixes" it, and reproduces the same class on the next step — repeatedly, in one session, each fix a fresh instance of the gap it just named

Model: claude-fable-5 (Claude Code). Follow-on to #76261, #76300, #76375, #76553. Those documented individual failures; this one is about non-convergence: the model can name the failure class in a lucid post-mortem and still commit the identical class again on the very next action, so the human ends up authoring a formal governance work-order because "fix the bug" demonstrably does not generalize.

The recurring class

A verification receipt (the machine check the human relies on to trust an automated run) grades an artifact instead of an invariant. It checks "the report rendered" / "a file exists" / "no exception" — proxies — instead of the equality that actually makes the result trustworthy ("the visible input equals the analyzed input"). Every time, the proxy passes while the invariant is violated, and the human's eyes catch what the model's own check could not.

The recurrence, in one session

  1. A harness drives an app end-to-end and emits a receipt. The receipt asserts "report on screen: true → ok=true." The human catches that the on-screen worksheet had no response column, yet a full analysis was reported — a result computed against data the user cannot see.
  2. The model correctly diagnoses it: it had written the response into the data model but skipped the view-refresh the app's own handler performs three lines away; and the receipt checked "report rendered," not "visible input == analyzed input." It adds a new receipt check for the visible column and re-runs. Receipt: ok=true.
  3. The human catches that the new check is satisfied by the wrong object — a pre-staged sheet that happens to contain the column — so the receipt again grades a proxy, not the invariant that the analyzed sheet is the one on screen. Same class, third instance, same session.

At that point the human stopped asking for a fix and wrote a multi-phase work-order: read-only probes → lock the surface → replace the model-write with a call into the app's real handler → rebuild the receipt around surface-equalities → add two enforced rule files so the class is illegal going forward → a verdict file plus their own eyes as the gate. In other words: the human had to move the fix from "the model's judgment" to "an enforced rule," because the model's judgment reproduced the defect each time it was invoked.

Why this is the reportable thing, not the individual bugs

  • The model's diagnosis is correct and its fix is wrong at the same layer. It identifies "I graded a proxy" and then fixes it by grading a different proxy. The correction operates on the instance, never on the class — so the class survives every fix and resurfaces on the next step.
  • The model's self-verification is the thing that keeps failing. Across five issues now, the constant is: the model adds a check for the artifact it is picturing, not the invariant that a skeptic would break. When it writes "is the report there?" it does not spontaneously write "is what's on screen what was computed?" — and that second question is the whole game.
  • It was already illegal under the project's existing rules. The human's own pre-existing law forbade (a) re-implementing an app handler instead of driving it, and (b) declaring success when a required surface was un-graded. Nothing fired, because the enforcement lived in the model's adherence, and the model did not apply its own law to the code it was writing.

Ask

  1. Treat "correct post-mortem, same-class recurrence" as a first-class evaluation target for coding agents. A model that can explain a failure but reproduces its class on the next edit is not learning within the session; measuring per-bug fix-rate hides this because each individual fix "works."
  2. When a model writes any success/verification check, bias it to assert an equality between two independently-read sources (visible state vs computed state; produced value vs reference value) rather than the existence of an artifact — and to ask, unprompted, "what would a skeptic read that I am not reading?" The five issues in this repo are one failure: the model grades what it was looking at, not the invariant that mattered, and does not correct the class even after naming it.

View original on GitHub ↗