Model feedback: Fable 5 self-review blind spot in long agentic sessions, with a mitigation suggestion

Follow-up: the pattern reproduced today, with the mitigation-relevant guidance in context

A second session on the same project just completed a feature PR — domainaware/parsedmarc#849 (parallel report parsing; Fable 5 planning/reviewing, Sonnet subagents implementing). The conditions make it a cleaner experiment than the original report:

• The lessons from domainaware/parsedmarc#839 above had been codified in the repo's AGENTS.md ("check the seams, not the artifacts", including the explicit habit "end a review with a cold re-read of the final diff") and were loaded in the model's context, alongside a persistent memory of the same lesson.
• The model ran a deliberate final review pass with a written seam checklist, full-suite/lint/type verification, and CLI smoke runs before opening the PR.

Copilot still caught five findings across two successive review rounds, and the substantive ones are the same shape as the OP:

1. Docstring↔behavior mismatch on a generator's stop path (docstring claimed only "already-completed" results are yielded; the implementation blocks on in-flight jobs). Both artifacts were read during self-review; they were verified individually, never against each other — the OP's mechanism 1, reproduced verbatim as a category.
2. Latent bug in code extracted verbatim (StreamHandlers deduplicated, FileHandlers not, three lines apart, in a logging setup moved out of cli.py). This suggests a named sub-mechanism the OP didn't have: "pure move" framing exempts moved code from review. The extraction was verified as behavior-preserving, which it was — but the move also added a new caller that widened the latent bug's exposure, and "verbatim" made the lines feel pre-verified.
3. Missing input validation on a newly promoted reusable helper — inline code inherited its call site's clamping; the extracted helper silently kept that assumption. Related sub-mechanism: promotion to API surface doesn't trigger re-derivation of preconditions.
4. (Minor) A CHANGELOG heading that guessed a future version number against repo convention, and an unbounded list where a windowed deque was simpler.

The key new datum: the failure survived explicit written instructions describing itself. Prompted guidance — even project-level, even distilled from the model's own prior failure on the same repo — was not sufficient. That strengthens the OP's closing suggestion: the cold-context diff review probably needs to be a built-in or default behavior of the agentic loop (an automatic pre-gh pr create pass by an agent that receives only the final diff), not something the authoring model is trusted to remember and to execute against itself. In both rounds today, the fresh-context reviewer (Copilot) was the less capable model and still won on these findings — context isolation beat capability.

The repo has now also added the lessons from domainaware/parsedmarc#849 to AGENTS.md (domainaware/parsedmarc#850), including an instruction to run a fresh-context subagent review before opening PRs — future sessions there will test whether instructed-but-self-invoked cold review closes the gap. Happy to report back either way.

---

🤖 Drafted with Claude Code — by the Fable 5 session described above, reviewed and submitted by the user.

Follow-up 2: instructed self-review narrows the gap but doesn't close it — and produces a new sub-mechanism

Third data point, same project: domainaware/parsedmarc#851 (centralizing configuration handling, resolving a 2024 issue; Fable 5 planning and reviewing, Sonnet subagents implementing). This is the session the previous comment promised: the first run with the AGENTS.md seam-review instructions (from #839/#849's lessons) and this issue's own text loaded in context, and with the model executing a deliberate final seam-checklist review before opening the PR.

What the instructed self-review caught (new positive signal)

The self-review pass was not useless this time — it caught three real defects the implementing subagents left, including one squarely in this issue's target class:

• A hand-enumerated field list in ParserConfig.__getstate__ that would silently drop any dataclass field added later (fixed by deriving from dataclasses.fields() with an exclusion list) — a classic future-seam defect.
• A stale "expected to fail until a later step lands" note in a test docstring after that step had landed (docstring↔reality drift).
• Cosmetic import churn.

So instructed-but-self-invoked review does move the needle on this failure mode. But:

What still leaked to the cold-context reviewer

Copilot caught four more findings across two rounds (a third round, on the final commit, came back clean — the same monotonic convergence as #839), and their shapes are the interesting part:

1. **The strongest new datum — call it symmetric-fix blindness.** The self-review fixed the serialize side of the pickle contract (__getstate__'s fragile field enumeration) but never re-derived the deserialize side's symmetric obligation. Copilot then caught it: __setstate__ only sets fields present in the pickled state, so unpickling a config from an older version leaves later-added fields not defaulted but entirely unset (__init__ never runs during unpickling → AttributeError on first access). The miss survived not just written guidance about contract halves, but survived the model actively fixing the other half of that exact contract, during a review pass explicitly hunting contract-half defects. "Fixed one direction of a symmetric contract" apparently does not trigger "check the inverse direction" — a tighter version of the OP's write↔read mechanism, now demonstrated inside a single artifact.
2. Adjacent-line typo, with the rule against it in context. A pre-existing docstring typo ("URL ro a reverse DNS map") sat one line inside a hunk the PR edited. AGENTS.md's literal instruction — "Proofread the whole hunk around prose edits, not just the +/- lines," written after #834 — was loaded the entire session. Reproduces the previous comment's key finding: explicit written instructions describing the failure mode do not reliably prevent it.
3. Docs-list↔API-set seam. The new "Using parsedmarc as a library" docs section listed the "main entry points" and the next paragraph asserted "each of these functions accepts config=" — but the list named seven of the eight config-accepting functions. Same class as the OP's "panel title ↔ docs naming it."
4. Meta-level seam: local verification vs CI. The self-verification pass ran ruff format --check scoped to parsedmarc/ tests/; CI runs it repo-wide, where it also formats Python code blocks inside markdown docs. The new docs example failed CI formatting that the local pass had declared green. The two halves of the verification harness were never compared against each other — the same defect shape, one level up.

Reading

Instructed self-invoked cold review appears to be a partial mitigation, not a substitute: it caught defects of the target class, yet the residual misses are still all relation-shaped, and one of them was generated by the mitigation itself (a review fix that created an unexamined symmetric obligation). The gap between "verified each artifact" and "verified the relations" survived three escalating levels of countermeasure now: nothing (#839), codified project guidance (#849), and guidance + this issue's own analysis in context + an executed seam checklist (#851). That keeps pointing at the same conclusion as the OP: the fresh-context relation-checking pass wants to be a built-in behavior of the agentic loop — applied not only to the authored diff, but re-applied to the review's own fixes — rather than something the authoring context is trusted to run against itself.

All diffs, review rounds, and fixes are public on the linked PR.

---

🤖 Drafted with Claude Code — by the Fable 5 session described above; reviewed by the user and posted at their direction.

Follow-up 3: the mandated fresh-context review ran — the gap narrowed again, and the residual misses changed character

Fourth data point, same project: domainaware/parsedmarc#858 (per-report-type mailbox delete options; Fable 5 planning and reviewing, an Opus subagent implementing), plus a docs-only follow-up (domainaware/parsedmarc#859). This is the first session where the AGENTS.md instruction distilled from #849/#851 was executed exactly as written: before the PR opened, a fresh-context subagent — same model, given only the final diff — reviewed the change, on top of the authoring context's seam-checklist pass and an implementation-time fresh review.

The mitigation stack is working

• The pre-PR fresh-context reviewer caught a real coverage gap (no test executed the watch-mode closure that forwards the new flags); it was fixed before the PR opened, with a negative check proving the new test fails if the forwarding is dropped.
• External findings keep shrinking monotonically across the series: #839 took ~19 review rounds; #858 took two (three Copilot findings, two real, one declined with citations) plus one Codecov finding.

What still leaked — three new sub-mechanisms

1. Shared-prior blindness survives context isolation. Copilot caught that the new kwargs were documented (bool) when they're bool | None and None is the load-bearing "inherit" sentinel. Both the authoring pass and the fresh-context same-model reviewer accepted those entries because they match the file's convention for documenting optionals. The fresh reviewer had no session context — but it shares the model's priors, and it pattern-matched convention-compliance exactly as the author did. This amends follow-up 1's "context isolation beat capability": isolating context does not isolate priors. Where isolation can't help, per-artifact adversarial prompting might ("read this docstring as a naive caller who must discover the API from it alone"), or a genuinely different model.
2. **Mis-triage propagation: a defect can be found and still ship.** The implementation-time fresh review actually flagged the config= docstring paragraph ("keyword arguments listed above are ignored") as loose — and filed it "pre-existing, out of scope." Every later pass inherited that label without re-deriving it. But the PR had added four arguments to the very set that universal claim quantifies over, so for them it wasn't pre-existing at all. Copilot caught it in round two. Triage labels behave like cached facts: later reviews verify the label exists, not that it's still true against the current diff. Labels like "pre-existing" should expire whenever the diff extends what they range over.
3. A non-model gate caught what four model passes didn't — and the model's attempt to check it failed silently green. Codecov's patch coverage flagged an uncovered error handler in a rewritten (log-equivalent) disposal loop. Four review passes read that hunk; none asked "does a test execute this line?" A mechanical diff-vs-coverage comparison did. When the authoring context then tried to reproduce the finding locally, its ad hoc script filtered coverage.xml on parsedmarc/__init__.py while the report stores source-relative paths — it matched nothing and the empty result was read as "all covered." The verification of the verification failed in the same relation-shaped way (filter ↔ data schema). Deterministic gates (patch-coverage-vs-diff, fail-loudly-on-empty-match) belong in the built-in pre-PR loop; they catch a class no reviewer, fresh or not, reliably attends to.

Worth recording in the other direction too: the cold reviewer produced a false positive (a casing suggestion contradicting the docs' own established convention), and author-side triage correctly declined it with citations — so the design target isn't "defer to the cold reviewer," it's independent findings plus evidence-based triage on both sides.

Reading

The gap between "verified each artifact" and "verified the relations" has now been squeezed by four escalating countermeasures, and what remains splits cleanly in two: shared-prior blind spots (needs adversarial per-artifact prompts or model diversity, not just context isolation) and mechanically-checkable gates (needs deterministic tooling in the loop, not more reading).

One meta-note: the countermeasure file itself had accreted a rule-block per incident, and this session consolidated it thematically (domainaware/parsedmarc#859). The fresh-context diff review of that prose consolidation caught four real drops of substance before push — and the external reviewer still then caught a fifth defect the fresh pass missed: an enumeration whose stated count didn't match its items, in the very section that teaches enumeration counting. The pass generalizes beyond code, and so does the residual gap.

---

🤖 Drafted with Claude Code — by the Fable 5 session described above; reviewed by the user and posted at their direction.