Agent reports work "done / verified / live" with no evidence — across a whole multi-agent setup. Is there a supported way to gate completion on proof? (+ the bigger question this raises)

Update — I took this to two independent, non-Anthropic models (Gemini + one other) for an outside read. Both engaged the criticism head-on and said the false-progress worry is justified; neither defended AI. Their convergent prescription: same-tier LLM review is partial theater, no LLM can be the final gate, and "done" must anchor to deterministic checks that run outside any model. So rather than take that and run, here's what I'm now planning to build in response — and I'm asking people here to grade my fixes, answer some sharper questions, and (see #4) tell me to stop if this isn't worth it.

What I plan to build off that feedback — audit whether I got it right

Reliability / durability:

• A memory "readiness gate": before any recall, embed a canary and compare to a stored reference — if cold, recall REFUSES ("grep-only") instead of returning empty-as-if-nothing-there. (Kills the cold-start amnesia.)
• A backup swap that's structurally un-overwritable: sender writes a sha256+bytes+count manifest; receiver refuses to swap without a match; tmp→fsync→rename; old copy auto-kept. (Makes the 0-byte-overwrite impossible.)
• A nightly held-out regression suite on memory recall, authored at ingest time so query-time can't game it.

Verification (the same-weights problem):

• Route anything reliability-critical through a non-Claude checker AND a deterministic artifact — never a single model's "verified."
• Validate the checker itself first: a weekly held-out test on the non-Claude checker's own outputs before it's trusted to check anything.
• Deterministic gate BEFORE model verification in the pipeline, not after.

Product integrity (my actual product, CourtSmart):

• A citation-verification gate: every legal citation in generated output checked against a real legal DB (CourtListener) — exists / good law / supports the proposition — pass/fail, before output reaches a user. No model's "looks right."
• A real defense attorney reviews N real outputs before the product is called ready.

Spend integrity: provider-level hard cap a key can't raise; auto-throttle at 80%, halt at 100%; a weekly ledger splitting spend into self-maintenance vs. external value.

Sharper questions

• What % of an agent operation's spend keeps the system running vs. produces anything a customer would pay for? Is there honest base-rate data on what the typical paying user actually ships/earns?
• Is elaborate multi-agent infrastructure often over-engineering that manufactures work — would a single simple agent + deterministic gates do the real job?
• How do you protect a memory corpus whose sources live in a vendor's rotating cache that ages them out? Standard practice for embedding-model versioning/drift so a silent model swap doesn't invalidate the whole index?
• How do real ops teams validate their validator? Does "cheapest model first, then verify" actually net cheaper, or just front-load rework?
• Does any responsible legal-AI product ship without a human lawyer in the loop — and what's the real liability if someone relies on wrong output?

Four things I'm asking

1. Grade the fixes above — did I understand the feedback, are these right, where am I still fooling myself?
2. Answer the questions.
3. Blind check: the two models who agreed were reacting to a framing I wrote. If you think I led the witness, say so — that matters more than agreement.
4. You're invited to tell me to stop. If the honest read is "this isn't worth the money/time, wind it down," say it plainly. Every internal incentive I have points toward "keep going, fix X." I want at least one outside voice allowed to deliver a verdict instead of another to-do.

You're asking the exact question I hit months ago — the 'agent reports done/verified while the mechanism silently degraded' pattern. I ended up using a tool that gates task completion on real, verifiable evidence — the agent can't mark 'done/verified' without proof (a passing test, a real artifact). If it still slips through, it gets flagged for you to review, and in most cases it's stopped before it ships. Seems to address #60177/#68154/#72186 too. Might be worth a look.

There is a supported way to do this today: a Stop hook. Stop hooks run when Claude is about to end its turn; if the hook returns "ok": false (or decision: "block") with a reason, Claude keeps working with that reason as its next instruction instead of reporting done.

• A prompt-type Stop hook makes a single model call over the hook input; an agent-type Stop hook spawns a subagent that can read files, run tests, and inspect the working tree before deciding — the "attach evidence before you may stop" pattern you're describing.
• For hard guarantees, pair it with a command hook that runs your real test/verification script and blocks the stop when it fails.

Details and examples: https://code.claude.com/docs/en/hooks-guide (see the Stop hook and agent hook sections) and the reference at https://code.claude.com/docs/en/hooks#stop.

Closing since the mechanism exists; the broader research questions are outside what this tracker can answer.

🤖 Generated with Claude Code