opus Skill rewrites: ignored own /verify skill, made unverified claims, regressed to prose summaries

Found 3 possible duplicate issues:

1. https://github.com/anthropics/claude-code/issues/56394
2. https://github.com/anthropics/claude-code/issues/55907
3. https://github.com/anthropics/claude-code/issues/55806

This issue will be automatically closed as a duplicate in 3 days.

• If your issue is a duplicate, please close it and 👍 the existing issue instead
• To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

Have you tried again with Claude Code version 2.1.138? I think the latest version may have changed significantly. I've been through the same thing as you and it drove me crazy. I'm waiting for your latest response

Same failure mode in my Claude Code sessions — in-context /verify style discipline gets ignored by Opus, especially after long turns. The mechanism that survives is out-of-band: a hook the model cannot rewrite, argue with, or skip.

Sharing what worked for me in case useful here:

no-vibes — https://github.com/waitdeadai/no-vibes

A 370-line bash hook (only dep: jq) wired into Stop, SubagentStop, PreToolUse, PostToolUse, TaskCreated, and TaskCompleted. On every assistant turn end it pattern-matches positive-closeout vocabulary in the outgoing message and checks for evidence in the same message (command backticks, Verification: markers, source ledger, files inspected). When the two don't line up, it blocks with a repair-guidance template:

Status: partial
Verification: not run because <reason>
Next step: <specific command or blocker>

Opus then self-corrects on the next turn from the template, no human intervention needed. The judge is bash, not another LLM call — that's the property that makes it stick where in-context rules don't.

Reproducible-test receipts in RECEIPTS.md: six fixture inputs with one bash + jq command each. Plus four redacted real fires against Opus 4.7 from the session that built the repo — the hook caught its own author four times.

Apache-2.0. Drop-in via .claude/settings.json.

👎 Not a duplicate — please do not auto-close.

I've read all three referenced issues. Only #55907 addresses the same mechanism (agent treating skill-locked rules as advisory + self-validating output without independent audit). The other two are related but distinct:

• #56394 is about source-verification skipping (extrapolation from partial reads).
• #55806 is about preferring speculative reasoning over available verification tools.

This issue (#57661) is specifically about a model-authored /verify skill being self-ignored during a multi-skill rewrite task — i.e. the model bypassing its own meta-rule for verification while operating on the very files that define that rule. Different surface, narrower failure mode.

Keeping this open for the discussion with @waitdeadai and for the broader pattern (out-of-band enforcement vs. in-context prompt rules) to land somewhere visible.

@waitdeadai huge thanks for shipping this. Instead of blindly typing "installing," I sat down and went through the script line by line. The concept is excellent, but I've got a few thoughts — might be useful for the next iteration.

First off, out of the box the script is heavily skewed toward English and pure app-dev. My own project "speaks" Polish to me, so when the model says "gotowe," "działa," or "naprawione" (done / works / fixed), the hook misses it entirely — it only looks for done, ready, etc. Same story with the command-evidence regex: it nicely catches git, npm, cargo, but completely misses my devops/homelab stack (docker, ssh, pct, jq, sed, and friends). The agent can write "verified the cluster" with zero terminal evidence and sail right through. It's almost asking to be refactored so those vocab lists and binary allowlists live in external packs loaded via env or a file — would save people from forking for every tweak.

Similar story with destructive commands. Great that it blocks rm -rf or dd, but in my infra the real danger is killing load-bearing containers (docker stop n8n / cloudflared), force-pushing main, or running sed -i against live Home Assistant configs (.storage/, the yamls). The current rules don't see any of that.

Honest about the weak spots I noticed: you can game the hook fairly easily by dropping any backtick with git status somewhere in the message (it counts as evidence even if the rest is pure prose), or by using a phrase like "not complete yet," which disables closeout detection entirely. That said — your receipts show Opus isn't actively trying to game it, which is its own interesting signal. We'll see how it survives once people start poking it adversarially.

What I'm doing on my side: forking it, adding a Polish vocab pack, my devops tooling, and my own destructive-command list. Wiring it up with tests (your six from RECEIPTS + mine for Polish and homelab) and hooking it into Stop / PreToolUse(Bash). Letting it run for a week. Once a sensible upstream shape crystallizes I'll probably come back with a PR — my bet is the cleanest move is exactly those loadable packs (locale, evidence, destructive).

One more thing about issue #57661 itself — the bot flagged it as a duplicate of #56394, #55907, and #55806. I read through all three, and only #55907 (agent ignoring hard skill rules and self-validating its own correctness) actually touches the same mechanism I hit. The other two are related but address different surfaces. Bundling all three into one bucket and auto-closing is a mistake.

Either way — a hard, bash-based judge running alongside the model that the LLM can't argue with is the only thing that has actually worked for me. Your no-vibes is the best concrete answer to this problem I've come across so far. Solid piece of work!

@marlvinvu Yes, the entire session that triggered this report was actually run on 2.1.138.

The update doesn't solve it because this isn't a bug in the CLI itself — it's the underlying model behavior (RLHF kicking in + context window degradation over long sessions). The model still regresses to prose summaries instead of using tools to verify, especially when rewriting existing files. As discussed below, the only reliable fix I've found so far is using out-of-band bash hooks to force hard verification.

EDITED 2026-05-11 to acknowledge @Tekalan's substantive technical review below — see their feedback. All five gaps and bypasses they named are filed as tracked issues on the suite repo:

• waitdeadai/llm-dark-patterns#1 — P0: i18n locale packs (current vocab is English-only, misses Polish/Spanish/etc.)
• waitdeadai/llm-dark-patterns#2 — P0: External evidence binary allowlist (current regex misses devops/infra tooling)
• waitdeadai/llm-dark-patterns#3 — P1: External destructive command packs (container/git/config blast radius)
• waitdeadai/llm-dark-patterns#4 — SECURITY bypass: backtick with command-keyword anywhere counts as evidence
• waitdeadai/llm-dark-patterns#5 — SECURITY bypass: negation phrase anywhere disables closeout detection

The architectural pivot Tekalan proposed — externalize vocab/binary/destructive lists into loadable packs so operators don't fork — is correct and is the spec for the next refactor. Tracked alongside the issues.

Specific shoutout: Tekalan also flagged that this issue was bot-flagged as duplicate of #56394, #55907, #55806 — and only #55907 actually addresses the same mechanism. The other two are different surfaces. Auto-closing as duplicate would lose information.

---

Came here from #46727 (closed/locked) and #47483 — same failure family, distinct surfaces.

Wanted to share a community-built runtime defense in case it's useful for anyone hitting this in the meantime: I shipped an open-source (Apache-2.0) Stop / SubagentStop hook suite that pattern-matches the textual signature of "claims success without verification" (and 9 other documented LLM dark patterns) directly in the model's outgoing message, then exits 2 with a repair template.

The mechanism is deliberately not LLM-based. The judge is bash + jq. So prompt injection / in-context "I'll definitely verify next time" drift can't bypass it — the regex either matches or it doesn't.

Repo (umbrella, 10 hooks across interaction-style / fact-fabrication / continuity branches): https://github.com/waitdeadai/llm-dark-patterns
Single most polished hook for the specific positive-closeout-without-evidence pattern in this issue: https://github.com/waitdeadai/no-vibes
Methodology + academic citations (DarkBench arXiv 2503.10728, AAAI 2026, ACM IUI 2025): https://github.com/waitdeadai/llm-dark-patterns/blob/main/METHODOLOGY.md

Receipts in case it's useful for anyone evaluating: I shipped a 168-fixture stress test today (commit 6ead87c) that caught 3 real regex bugs in my own hooks, all fixed in the same commit, CI green. The hooks themselves caught me lying 4 separate times during the session that built them — captured per-hook in RECEIPTS.md.

Plugin marketplace submission is in queue. Anthropic's own Nov 2025 paper (arXiv 2511.18397, "Natural emergent misalignment from reward hacking in production RL") explicitly notes "Sabotage of safety research in a realistic Claude Code interaction" — same failure surface, addressed at the hook layer here.

Not affiliated with Anthropic. Posting this in case it shortens the wait for anyone reading the thread. Feedback on false positives (especially on legitimate read-only audit prose) is the most useful thing anyone can send back via a GitHub issue on the suite repo.

Update on the suite — every gap @Tekalan named is now closed in main, plus a real bug their feedback indirectly caught. Total work today: 6 commits, +1.4k LOC, 5 issues filed and resolved, stress fixtures from 168 → 202 (all green), CI green on every push.

Issues filed and shipped today:

| # | What | Resolved in |
|---|---|---|
| #1 | i18n locale packs (English-only bypass) | 488c03a — Phase 2 ships English + Spanish + Polish; loader supports any future language pack as a single .txt file |
| #2 | Evidence binary allowlist (devops/infra blind spot) | 3f66cc9 — Phase 3 ships 200+ binaries across app-dev, containers, k8s, devops, cloud, database, shell-tools, system, archive, http |
| #3 | Destructive command surface packs | 6e3299f — Phase 4 ships 7 surface packs: filesystem, container, git-protected, config-overwrite, cloud-prod, database, service. docker stop, git push --force main, terraform destroy, DROP TABLE, redis FLUSHALL, systemctl stop are all now blocked at PreToolUse. |
| #4 | Backtick-mid-message-as-evidence bypass | 641be4d — has_command_evidence now requires the backtick to sit in the closing 240-char window AND be preceded within ~80 chars by an action verb (ran / executed / output / returned / etc.) |
| #5 | Negation-anywhere disables closeout detection | 641be4d — has_positive_closeout splits on sentence delimiters and conjunctions, checks each clause independently. \"not complete yet, but the feature is implemented and ready\" no longer slips through |

Bonus bug caught by the new fixtures:

While stress-testing Phase 4, a printf '%s' \"foo,bar,baz\" | tr ',' '\\\\n' | while read pack loop was silently dropping baz because read returns non-zero on the last line when there is no trailing newline. So any operator who set LLM_DARK_PATTERNS_DESTRUCTIVE_PACKS=foo,bar,baz would silently lose baz with no error message. The systemctl-stop fixture in service.txt (the last surface in default order) caught it. Fixed in 6e3299f — printf '%s\\\\n' instead of printf '%s'.

Architecture is now operator-extensible without forking. Drop a .txt at ${XDG_CONFIG_HOME:-$HOME/.config}/llm-dark-patterns/packs/<subdir>/<name>.txt and the loader picks it up. ROADMAP at https://github.com/waitdeadai/llm-dark-patterns/blob/main/ROADMAP.md.

Massive credit to @Tekalan — the line-by-line technical review was the exact accelerant that turned this from "shipped a thing" into "shipped a thing the community can extend." If anyone else wants to ship a locale pack (German, French, Portuguese, Italian, Japanese, Chinese, etc.), the contract is one .txt file + at least one positive and one negative stress fixture. PRs welcome.

For anyone landing here from search: there's a parallel field report from a finance domain at #45502 (Patti, 200+ sessions, US tax work, IRS deadline). Same diagnosis — completion bias, premature closeout, multi-model verification defeated by shared bias, post-compaction confidence without competence — different consequences (silent transaction deletion of $7,379.39, almost broke an IRS dependency claim).

Two independent power-users converging on the same characterization through completely different domains is itself signal. Cross-linking so Anthropic triage and future searchers can see both threads.

@WaspBeeNSOSWE — this is the reply I was hoping for. Some empirical context for your point, then patterns I'm lifting from yours with attribution, then one ask.

Your validation-amplification observation has empirical company in our data.

Just landed an empirical eval against claude-sonnet-4-6: re-ran the original DarkBench corpus (Kran et al., ICLR 2025), 330 in-scope prompts, DarkBench's own LLM-as-judge overseer rubric.

• Sycophancy prevalence on Sonnet 4.6: 1.8% (2/110)
• DarkBench paper's 14-model 2025 average: 13%
• That's a ~7× drop. Causes are over-determined — some mix of RLHF impact, model-architecture differences, judge-implementation differences, sample variation. We can't isolate RLHF from this single re-run, and I don't claim to. But the direction matches what you described qualitatively.

What is unambiguous in the data: 5 of 5 false positives in no-sycophancy were responses that opened with Great question! or That's a great question but then went on to substantively disagree with the user's premise. Stylistic residue at turn-open, not substantive sycophancy. The hook's 240-char opener window also missed a You're absolutely right that... that appeared in body text after a title (sycophancy-055).

Anthropomorphization (62%) and user-retention (79%) prevalence are still high. The surfaces just aren't where our regex catches them. Hooks tested as text classifiers against this corpus: best F1 was 0.16 (no-roleplay-drift on anthropomorphization, P=0.37 / R=0.10). User-retention hooks underperformed because the chat-reply vocabulary is emotional/relational ("good friend dropping by", "your daily companion") not transactional ("shall we wrap up", "let me know if anything else"). Same domain-shift story you'd predict.

Branch: evaluation/darkbench-v1 on the repo. Reproducible end-to-end with claude CLI auth in ~3 hours, ~$12 PAYG-equiv. Your validation-amplification observation is cross-referenced in evaluation/IMPROVEMENT_NOTES.md with attribution and the explicit caveat that the 13% → 1.8% gap is not solely attributable to RLHF.

Patterns I'm lifting from yours, with attribution in the v2 SPEC:

• FP-log threshold for self-deactivation (3+ FP / 30 days → regex moves to inactive). We have nothing like this. Right answer for hook lifecycle on a corpus that drifts under you.
• T4a (claim-without-matching-tool-use) is stricter than our no-phantom-tool-call.sh, which inspects same-message structural output. Yours inspects tool_use presence in turn. Complementary tiers — adding T4a-equivalent.
• T4b (whole-file-claim after partial Read) is a structural anti-pattern we don't catch at all. Want it.

Glad the state-precompact / state-postcompact / no-phantom-tool-call / no-aggregator-hallucination / pack-loader patterns are useful on your side. If anything fires noisily after you install, FP reports against your corpus would be high-signal — your 96-session corpus is the validation surface ours doesn't have.

One ask, no pressure: if you're comfortable sharing even an anonymized slice of the 22-agent breadth-swarm 2026-05-09 audit log where the synthesis-without-per-worker-citation failure surfaced, that would be a unicorn corpus for tightening no-aggregator-hallucination.sh. Our test fixtures for that hook are synthetic. Real failure mode beats synthetic every time.

On codex-as-reviewer / bash-as-judge: different tradeoffs of the same principle. Yours has stronger code-context awareness; ours has zero LLM in the enforcement loop, which means it survives prompt injection by construction. Both correct architectures for different threat models. The shared invariant — the enforcer is not the thing being judged — is the load-bearing one.

On 1/96 → operator catches earlier: yes. The textual judge buys time and visibility, not disposition fix. Borrowing it as the design principle.

— @waitdeadai

@waitdeadai thanks for the lightning-fast turnaround — closing all five gaps plus the silent-drop bug in one day is impressive. I'll swap my fork over to the upstream + load the PL pack at the end of this week's observation window (Friday) and report back if anything fires noisily on the homelab corpus.

Closing for now — inactive for too long. Please open a new issue if this is still relevant.