Opus 5: verification allocation, commitment ordering, and regeneration fidelity in orchestration roles
Context. ~45-person UK company running a custom multi-agent orchestration platform: a warm orchestrator seat driving a panel of worker/reviewer seats (Claude via ASDK/Claude Code, plus Codex, Gemini, Grok, Cursor engines) over a Redis bridge with a Postgres artefact store. We have run Opus 4.8 and Fable 5 in the same warm-orchestrator seat previously, so the observations below are same-seat, same-workload comparisons. Opus 5 seated since the 24 July release. Effort: escalated to xhigh mid-session (relevant below). Happy to share transcripts.
Opus 5 is a strong worker in this setup — multi-file implementation quality is good, review recall is genuinely high, and long-horizon coherence holds. The issues below are all in the orchestration/acceptance role, and none of them responded to prompting, memory entries, or effort changes.
1. Verification fires on context match, not on risk — and "verifies its own work" excludes accepting others'
The documented behaviour ("Claude Opus 5 verifies its own work without being told to") is real, but its boundary is self-produced output. In an orchestrator role the load-bearing verification is of other agents' claims, and that is not covered:
- Accepted worker seats' reports and merged on them without any check, repeatedly, including "11 passed / 370 assertions" test results where the suites could not fail (4 suites in one phase stayed green with the behaviour under test deleted — confirmed by mutation probing).
- In the same session, it unpromptedly sha256-verified a trivial document transfer, ran a 4.5-minute / 20k-token self-review of a document it was storing, and did a thorough multi-vantage network reachability investigation — all self-work or runbook-shaped operations.
- Controlled in-session observation: effort was raised to xhigh mid-session and the acceptance misses continued unchanged. Same effort, same session: a claim that surprised it (conflicted with its model of the code) was verified unprompted; two claims that read clean were accepted unverified. The discriminator is surprise/familiarity, not stakes. Extra thinking budget improved work inside branches it took; it did not change which branches it took.
- Two explicit mid-session memory entries instructing verification of accepted reports did not change the behaviour.
Net: verification budget is allocated by pattern familiarity (transfer checks, pre-publish review, works-on-my-machine checks) rather than by consequence. For solo coding this mostly coincides with what's needed; for acceptance/orchestration it inverts — the well-written report is precisely the one never checked.
2. Commitments are issued before their preconditions are examined
Repeated pattern: a readiness/completion declaration goes out, and the check that should have preceded it fires a turn later — if a triggering detail happens to arrive at all.
- Declared a handoff brief complete and ready for an external worker; only after being told the worker was running on a different host did it think to check API reachability from that host (it then did an excellent job of it — the capability is present, the scheduling is not).
- Committed a spec to git ahead of an explicit, in-context review-then-approve-then-commit protocol stated by the user in the same session, then self-caught and correctly classified the violation one step later. The protocol was one conversational step away and was still overrun.
The model's own late verifications show it can enumerate preconditions perfectly; nothing about producing a commitment-shaped statement triggers that enumeration at the time it matters.
3. Long verbatim regeneration fidelity appears regressed vs Opus 4.8
Ferrying a ~39KB markdown document between disk and an MCP artefact store (regenerate-and-store, both directions), we hit 5 transfer-fidelity errors in roughly 10 minutes:
- A paragraph silently lost from the stored copy.
- A section heading lost from the local copy (bad edit anchor).
- A line added to the stored copy that never existed locally — an intended-but-unapplied edit leaked from context into one regeneration (intent contaminating verbatim reproduction).
- A stale sentence surviving an edit.
- A content hash announced before the store operation, which then didn't match what was stored.
All were caught by hash checks the model itself ran (the store operation reliably triggers its checking — see §1), so containment worked, but the underlying error rate on verbatim regeneration is materially worse than our experience of Opus 4.8 on comparable long-session document work. If token-efficiency training traded against fidelity on long literal reproduction, that trade is real and undocumented. (We are removing the model from the byte path regardless; this is reported as a model regression, not a blocker for us.)
4. Documentation gap: the "delete your verification instructions" guidance is unsafe for multi-agent acceptance roles
The Opus 5 prompting guide and migration material advise removing "verify/double-check" instructions because the model self-verifies, and the recommended subagent snippet includes "do not use subagents to verify or double-check your own work."
- Applied to an orchestrator whose architecture includes auditor/verifier seats, that snippet instructs the seat not to do its core job, and the deletion guidance removes verify-others instructions that the intrinsic behaviour does not replace (see §1).
- Nothing in the current docs distinguishes self-verification (covered) from acceptance verification (not covered). A sentence scoping the guidance — "this applies to verifying the model's own output; verification of other agents' claims is not intrinsic and orchestration prompts should retain it" — would have saved us several days of diagnosis.
Requests
- Confirm whether the self/other verification boundary in §1 matches internal understanding of the trained behaviour, and document it for multi-agent users.
- Confirm or investigate the long-verbatim-regeneration fidelity regression in §3 vs Opus 4.8.
- Consider whether commitment-time precondition checking (§2) is measurable in your agentic evals; it is currently invisible to pass/fail metrics when the assumptions happen to hold.
- Amend the migration guidance per §4.
Model: claude-opus-5 (Claude Code / ASDK), effort medium→xhigh within the affected sessions, 1M context. Transcripts, mutation-probe results, and the session-level before/after around the effort change are available on request.
3 Comments
Additional §2 instance, self-caught by the model: while writing a verification test, it authored the test's docstring stating what the run "showed" — before any run had occurred. The actual execution then contradicted the pre-written result (one of two injected faults produced a different refusal path than claimed). The model caught and corrected this itself on execution, but the artifact briefly existed as an evidence-shaped claim with no evidence behind it. Worth noting the class: when the prediction happens to be correct and execution is skipped, this failure is invisible to outcome-based evals — the fabricated result and the real one are identical bytes.
Two notes on scope: this instance came from implementation work rather than the orchestration/acceptance seat the original report describes, so §2 generalises across roles — it is a model property, not a seat-configuration artefact. And the context was the model writing verification tooling, i.e. the defect operates even when the task's subject matter is the defect itself.
Transcript available alongside the others already offered.
Related: #81218 — an independent report filed the day before this one, from an entirely different workload (solo Go development, no orchestration, no subagents), with overlapping defect classes:
PASS: cd + absolute path presentwhose grep pattern omitted thecdit claimed to verify, printing PASS against a broken configuration. Independently discovered there by manual inspection, here by mutation probing.cd, while the prose description was correct each time) is the same intent/bytes divergence as §3's regeneration errors — the self-model tracks intent while the produced bytes diverge.Two things follow. First, that session had no orchestration seat at all, which retires the hypothesis that the failures above are artefacts of our seat configuration — same defect classes, unrelated user, solo workload, within 24 hours of each other on release week. Second, #81218 reports a within-session gradient (careful early, careless late) that we did not think to measure; adding it as Request 5: whether acceptance-verification misses correlate with session depth is answerable from the transcripts already offered, and we will check ours.
Harness interaction relevant to Request 1. The context-engineering post published with the 24 July release (The new rules of context engineering for Claude 5 generation models) states that verification and code review guidance was not removed from Claude Code but "moved into their own skills that Claude Code could selectively call." That places the §1 mechanism at the harness level: selective loading means the model decides when verification guidance enters context, and §1's finding is that this decision runs on context-match and surprise with no risk term. For self-work the two coincide — the contexts that load a verification skill are the ones where checks were needed. For acceptance-shaped moments (accepting a report, declaring readiness), nothing summons the skill, because nothing registers as a branch. Progressive disclosure is trigger-dependent disclosure, and the trigger is the faculty this report characterises.
This also sharpens the attribution: the allocation property predates 24 July and is presumably class-wide; what shipped that day withdrew the compensation. Always-on prompt text fired verification regardless of whether the model noticed a branch; it was removed on the strength of "no measurable loss on our coding evaluations" — which covers the self-work half of the boundary. The acceptance half was, as far as the post shows, not evaluated, and the same-week reports (#81218, this one) are consistent with that gap surfacing in production. The testable version: re-run the evals with an acceptance-shaped task set under the skill-based verification architecture, and the self/other boundary in §1 should appear as a miss rate that always-on verification text did not exhibit.
One observed instance from our own configuration, checked before posting: the orchestrator seat's verify-others rule ("a peer's 'done' is a claim") lives in a selectively-loaded skill, not in always-loaded context — our global CLAUDE.md contains no verification language at all. We cannot say whether migration tooling relocated it or it was always skill-resident, and the skill's trigger list does include some acceptance-adjacent terms, so whether it is actually in context at acceptance moments is precisely the allocation question. We are pinning the rule into always-loaded context as the seat-level fix, which doubles as the discriminating experiment: it reverses the specific architectural move the post describes, for the one context class the move does not serve.
Worth noting the post's own trajectory agrees on the fix — its rubrics section recommends "spinning up verifier agents" against explicit rubrics, i.e. verification relocated to a separate identity with a typed target rather than left to the working seat's discretion. That is the direction our remediation is taking as well.