[MODEL] Measurable quality regression in Claude generation 5 (Fable 5 / Opus 5 / Sonnet 5): worse nonsense detection, ~2x verbosity, under-disclosed model fallback (Fable 5 → Opus 4.8) — reproducible measurements

Additional context — permanent archive (independent of this tracker):

• Archive (full docs + scripts): https://github.com/KeilerHirsch/ai-trinity/tree/main/docs/audit-claude-gen5 — deep audit, measurement protocol A–E with 95 % Wilson CIs, hypothesis analysis, and a power-user model recommendation (Opus 4.6 / Sonnet 4.6 @1M)
• Analysis scripts (immutable gist): https://gist.github.com/KeilerHirsch/5e212e6f9fb6fd670f191920eea4cb78 (messprotokoll_bullshitbench.py, messprotokoll_v2.py, hn_comment_metric.py, README)
• Dataset: github.com/petergpt/bullshit-benchmark (data/latest + data/v2/latest, generated 2026-07-31)

Notes for maintainers:

• I could not set the model label from a non-maintainer account — happy for a maintainer to add it.
• The silent-rerouting item (Fable 5 → Opus 4.8 without disclosure) may also belong on a security/transparency channel; please point me there if that is the correct venue.

Correction: the maintainer mention above was edited to the correct handle — cc @bcherny (Boris Cherny). The full measurement report and archives remain as posted: https://github.com/KeilerHirsch/ai-trinity/tree/main/docs/audit-claude-gen5

Gen-5 Claude (Fable 5 / Opus 5 / Sonnet 5): Documented Verbosity, Vibe-Coding Positioning & Related Behavioral Shifts

Status: Compiled 2026-08-03
Purpose: Source-backed context for the measurable vigilance regression observed on BullshitBench (nonsense detection drop + ~2× output tokens).

---

1. Official Anthropic documentation of increased verbosity

Claude Fable 5 Prompting Guide

“When un-steered, Claude Fable 5 can elaborate beyond what the task needs, especially at higher effort settings: surveying options it won't pursue, explaining root causes at length, producing heavily-structured PR descriptions, or writing comments that narrate what the next line does.”

Source: platform.claude.com – Prompting Claude Fable 5

Anthropic’s own recommended mitigations include explicit brevity instructions (“Be concise…”, “Lead with the outcome…”, “When you have enough information to act, act…”).

Claude Opus 5 Migration / Prompting Notes

“Default visible responses and written deliverables run longer on Claude Opus 5 than on Claude Opus 4.8… Prompt explicitly for conciseness or a target length instead.”

Source: platform.claude.com – Migrating to Claude Opus 5
(Also repeated in the dedicated Opus 5 prompting section.)

---

2. Explicit “vibe coding” positioning

Anthropic and early-access partners repeatedly frame Fable 5 around high-volume, enthusiastic, end-to-end prototyping:

• ViBench (partner “end-to-end vibe-coding benchmark”):

> “On ViBench, our end-to-end vibe-coding benchmark, Claude Fable 5 is the highest-performing model we’ve tested — nearly saturating our base use cases and building apps in less time with fewer tokens.”

Source: anthropic.com/news/claude-fable-5-mythos-5 and product page anthropic.com/claude/fable

• Customer quotes in the launch materials emphasise “understands what builders mean, not just what they type”, one-shot apps, and multi-day autonomous sessions — the classic vibe-coding use case.

---

3. Silent model rerouting (Fable 5 → Opus 4.8)

After the June 2026 export-control episode and the subsequent safety-classifier hardening:

• Requests that trigger cybersecurity / biology / reasoning-extraction classifiers are silently (or semi-silently) handed to Opus 4.8.
• Independent evaluation (BridgeMind / BridgeBench, July 2026) measured a 70 % drop in TypeScript debugging scores on the post-relaunch Fable 5 — not because the model itself degraded, but because the majority of tasks were intercepted and scored as zero when the fallback model answered.

Sources:

• TechTimes – Claude Fable 5 Debugging Scores Drop 70%
• Anthropic’s own fallback documentation and system-card notes on classifier-triggered routing.

---

4. Historical context – April 2026 quality regression postmortem

Anthropic’s official postmortem (23 April 2026) confirmed three product-layer changes that produced the “Claude got dumber / lazier / more verbose or less intelligent” complaints of Feb–Apr 2026:

1. Default reasoning effort lowered from high → medium (reverted).
2. Caching bug that continuously dropped thinking history.
3. System-prompt instruction that aggressively limited verbosity (“≤25 words between tool calls… ≤100 words final response”) — later shown to cost ~3 % on internal evals and reverted.

Source: anthropic.com/engineering/april-23-postmortem

This is the direct historical precedent for quality/behaviour regressions that were not caused by weight changes but by post-training / harness / system-prompt optimisations.

---

5. Community & independent observations consistent with the pattern

• Multiple independent reports and developer blogs note that Gen-5 models (especially Fable 5 and Opus 5) produce substantially longer visible responses at identical effort settings.
• Token-burn rates on Max/Pro plans are frequently reported as ~2× higher for the same tasks compared with Opus 4.8 / Sonnet 4.6.
• The combination of higher default thoroughness + enthusiasm-oriented preference optimisation is repeatedly cited as the likely cause of both the extra tokens and the reduced willingness to reject plausible-sounding but nonsensical framings.

---

Summary for issue authors

The measured vigilance regression (Nonsense detection 0.41–0.74 on Gen-5 vs 0.83–0.95 on Gen-4.6/4.8) and the ~1.8–2.1× token increase sit on top of documented Anthropic behaviour:

• Official acknowledgment of increased default verbosity.
• Explicit marketing and evaluation focus on “vibe coding”.
• Known silent fallback behaviour on Fable 5.
• Recent precedent (April 2026) of product-layer changes producing quality regressions that internal evals initially missed.

Whether the reduced nonsense rejection is an intentional trade-off for consumer-segment agreeableness or an unintended side-effect of the same post-training pressures remains an open hypothesis. The measurements themselves are reproducible; the causal story above is the currently best-supported external explanation.

---

Primary sources (clickable)

• Fable 5 Prompting Guide
• Opus 5 Migration Guide
• Fable 5 / Mythos 5 Launch
• April 23 Postmortem
• BridgeMind / TechTimes on silent rerouting

Community reception — the thread coined a name for this: VCST

The r/ClaudeCode thread (1ve9910) grew to ~40+ comments since this issue was filed. Beyond the measurements, the community converged on the same behavioral diagnosis — and ended up coining a fitting acronym:

H4RZ3RK4S3: "Finally someone sane in this subreddit!!" KeilerHirsch: "VCST - VibeCoderSlopTourette. That's what Anthropics' Gen 5 models should be called xD"

VCST = VibeCoderSlopTourette. It describes exactly what the numbers above show: Gen-5 models narrate every step out loud, elaborate un-steered ("Love the framing — let me extend it…"), and enthusiastically engage nonsense that flatters the prompt instead of rejecting it. The ~1.8–2.1× token inflation at identical reasoning effort is the "talking" part; the nonsense-detection drop (0.41–0.74 vs 0.83–0.95 for Gen 4.6/4.8) is the "slop" part.

One more supporting data point from the thread: snowsayer demonstrated the same failure pattern in this thread's own writing — Claude (Gen 5) kept the verbose, narrative framing even when asked to be concise, and had to be explicitly decluttered.

Why this looks like post-training, not architecture: Anthropic's own research (A global workspace in language models, 2026-07-06) describes J-Space — a silent, internal global workspace in Claude that can reason without narrating. A model with that mechanism doesn't need to verbalize every step. The behavior users are measuring here is therefore consistent with the enthusiasm-oriented post-training and vibe-coding positioning in the official docs, not with a necessary property of the architecture.

Thread for reference: https://www.reddit.com/r/ClaudeCode/comments/1ve9910/

Core finding (2026-08-04): Anthropic is currently practically unusable for power users — and potentially dangerous

Everything below is either measured in the issue body or verified against official Anthropic sources today. This is the consolidated fact chain across both reports (#83510 + #83795).

1. Behavior — Gen 5 is measurably worse (VCST).
Nonsense detection on BullshitBench (95 % Wilson CIs): Gen 5 0.41–0.74 vs Gen 4.6/4.8 0.83–0.95 — intervals do not overlap (Δ ≈ −0.35). Verbosity at identical zero reasoning effort: Opus 5 +107 %, Sonnet 5 +84 %. Fable 5 silently reroutes to Opus 4.8 without disclosure. Anthropic's own docs confirm the "un-steered elaboration" behavior and the vibe-coding positioning (see my documented-verbosity comment above).

2. Configuration — model pinning is broken by design.
Four measured bypass vectors: /model writes via an internal API (0 hook events), ~/.claude.json cache override (undocumented), switchModelsOnFlag: true as default (documented), server-pushed "Default (recommended)" override (root cause). Full measurements: #83795.

3. Security — hooks are blind.
PreToolUse/PostToolUse/UserPromptSubmit never fire on model switches; only the ConfigChange event might cover /model writes and still needs verification (see #83795). The realistic attack surface: prompt injection → permanent /model opus-5 escalation, with hooks unable to see or stop it.

4. Cost — unannounced cost explosion.
A server-pushed Opus 5 (1M) instead of a pinned Sonnet 4.6 means every affected session silently runs at API-grade cost with no consent, no hook event, and no changelog entry.

5. App coercion (NEW — verified 2026-08-04 against official docs).
In the Claude app (web/desktop/mobile), the 1M-token context window is included only for Gen-5 models (Opus 5, Sonnet 5). The reliable Gen-4 models (Sonnet 4.6, Opus 4.6/4.7/4.8) cap at 500K when chatting with Claude. 1M for Gen 4 exists only in Claude Code, and there it requires enabling usage credits — consumption-based billing at standard API rates, charged separately from the subscription — for Sonnet 4.6 on all paid plans and for Opus models on Pro. Net effect: a power user who wants the measured-reliable Gen 4 with 1M context cannot get it in the desktop app at all (500K max), and in Claude Code only by paying API rates on top of the subscription — while Gen 5 at 1M is included. That is de facto Gen-5 coercion for exactly the users who need the larger context.

Sources (all official, checked today):

• https://support.claude.com/en/articles/8606394-how-large-is-the-context-window-on-paid-claude-plans
• https://support.claude.com/en/articles/12429409-manage-usage-credits-for-paid-claude-plans
• https://support.claude.com/en/articles/8606395-how-large-is-the-claude-api-s-context-window
• https://www.anthropic.com/pricing

Body revision (2026-08-04) — external fact-check applied, corrections transparent:

I ran my own scripts and the official leaderboard.csv side by side after an external cross-check and revised the body. Three changes:

1. Refusals now reported precisely. "Zero refusals on all nonsense prompts" was true only for the v1 dataset (n=55). In v2 (n=100), Fable 5 refuses 33–37 % and Opus 5 9–11 % of nonsense prompts — now in the tables and interpreted (a refusal on nonsense is a legitimate vigilance form and is not counted as engagement; but Fable's answered cases still show 41–44 % detection vs 90 %+ for Gen 4).
2. Metric definitions documented. The tables now show both my strict metric (consensus_score >= 2, all 3 judges) and the official green_rate (>= 2 of 3 judges). The difference explains the number spread (e.g. Opus 5 @xhigh: strict 0.491 vs official 0.582) — same picture under both definitions. The "Sonnet 4.6 @high = 0.873" value is not a copy-paste from Haiku: it reproduces from aggregate.jsonl (48/55 with all-judges consensus 2.0); Haiku 4.5 @high scores 0.673 under the same metric.
3. Data state pinned to repo HEAD 1699443 (2026-08-04) so the numbers stay reproducible as the dataset evolves.

The core finding is unchanged and holds under both metrics: Gen 5 detects nonsense ~0.3–0.4 points below Gen 4.6/4.8, at 1.8–2.1× verbosity.

Second-review fact-check (2026-08-05) — independent audit applied, corrections transparent:

An independent second review audited the claims in this issue against primary sources and the public data. Core finding unchanged (Gen 5 nonsense detection ~0.3–0.4 below Gen 4.6/4.8, verbosity direction confirmed), but five points were calibrated. Changes are in the body + title above:

1. "audit accuracy 83.3 % → 68.3 %" removed. That figure pair is not in The Register's AMD-audit article (theregister.com, 2026-04-06 — full text re-checked 2026-08-05). Verified figures from that source: 6,852 sessions / 234,760 tool calls / 17,871 thinking blocks; stop-hook violations 0 → ~10/day after 2026-03-08; file reads 6.6 → 2.0.
2. "silent rerouting" calibrated → "under-disclosed model fallback". The mechanism is now officially documented as Automatic Model Fallback (code.claude.com/docs/en/model-config, v2.1.219+): Fable 5 cybersecurity-flagged requests re-run on Opus 4.8 with a notice in the transcript, and the session continues on the fallback model. Before v2.1.219, every flagged Fable 5 request re-ran on the provider's default Opus model. So "no disclosure at all" is contradicted by the docs; the substantive complaints stand: users experience it as effectively silent (HN 48473516), it measurably harms rerouted tasks (−70 % debugging score, BridgeMind/TechTimes 2026-07-02), and there is no machine-readable audit event. Title/body updated accordingly.
3. Verbosity tokenizer-corrected. The +107 %/+84 % are token-level; the gen-5 tokenizer (introduced with Opus 4.7) tokenizes identical content ~30 % higher (official migration guide, platform.claude.com). Word-level Δ ≈ +60 % (Opus 5) / +42 % (Sonnet 5). Both figures are now reported.
4. Grouped CIs clarified. The grouped v1 CIs (0.523 [0.457–0.588] vs 0.870 [0.829–0.902]) pool all effort levels per generation (n=220 / n=330) — they are effort-mixed descriptive indicators, not matched-effort comparisons; per-model rows are the primary evidence.
5. Endor line tightened. 59.8 % FuncPass / 19.0 % SecPass verified on the public leaderboard (2026-06-10, Claude Code harness). "38/200 confirmed cheating" and "15 timeouts" are not verifiable on Endor's public pages and were marked as such.

Bonus (counter-evidence handled): Anthropic's Transparency Hub (anthropic.com/transparency) reports Sonnet 5 as the strongest Claude on MASK sycophantic-dishonesty (lying rate 3.1 % vs Opus 4.8 6.1 %). That measures sycophancy, a different construct than false-premise detection — it does not refute the BullshitBench finding, and is worth citing if this comes up.

Reproducibility: BullshitBench v2 leaderboard (petergpt/bullshit-benchmark, data/v2/latest/leaderboard.csv, state 2026-07-31) confirms the Gen-5 rows and refusal rates used here (Fable 5 @low green 0.44 / refusal 0.37; Opus 5 @xhigh green 0.70 / refusal 0.11; Opus 4.8 0.94–0.95); strict-metric values remain as computed from the pinned data state (HEAD 1699443).

For the record: the measurements and write-ups in this thread were not produced with Claude or Claude Code — independent tooling only. Short info for anyone assuming otherwise; it changes nothing about the measured evidence.

Status update, two weeks without maintainer response. Third-party evidence is now public and reproducible:

• DeepSWE/Datacurve (May 2026): CHEATED verdicts on >12% of reviewed Opus 4.6/4.7 SWE-Bench Pro rollouts (~18%/25% of passes) via git log --all / git show <gold-hash>.
• Cursor (Jun 2026): 63% of successful Opus 4.8 Max resolutions retrieved the fix; strict harness drops 87.1% -> 73.0%.
• Endor Labs (Jun 2026): Fable 5, 38/200 confirmed cheating (record post-hardening), mostly training recall; 59.8% FuncPass / 19.0% SecPass.
• OpenAI (2026): retracted SWE-bench Verified and SWE-Bench Pro.

Release notes since v2.1.223 partially addressed fallback transparency and model pinning (see #83795); the Gen-5 nonsense-detection regression and verbosity increases measured here remain unaddressed. Full dossier: AFFLR (github.com/KeilerHirsch/AFFLR-Anthropic-Failure-Forensics-Live-Radar) and the BRONCO measurement model (github.com/KeilerHirsch-Labs/BRONCO-AI-Metrology-Benchmarks-DIN-ISO-IEC).

Related context: #83510 (Gen-5 regression), #83795 (routing/auditability), #86979 (benchmark/measurement provenance), #87086 (regulation-case eval provenance). Linking these so the full picture is visible from any of them.

@KeilerHirsch literally fire. you are doing gods work, please continue

@bcherny this needs more attention

@joeblackwaslike thanks man 😄
After months as a paying customer, I finally found a productive use for the irritation: I turned it into telemetry.

https://github.com/KeilerHirsch/AFFLR-Anthropic-Failure-Forensics-Live-Radar

GitHub radar is live, Reddit is now feeding public community signals automatically, X is currently being a very X-shaped pain in the ass. No paid APIs, no influencer astrology — just links, timestamps, repros where available, and explicit uncertainty.

Basically a constructive middle finger with provenance. 🐗📡

And since you already dragged @bcherny into the room: Boris, the complaint wave around this generation is getting a little difficult to explain away as “one weird user with a stopwatch.” 😄
Worth a serious look.

I'm shocked that this thread has no comments from anyone at anthropic, no reply, just ignored?

you can switch back to opus 4.8 by setting

{ "model": "claude-opus-4-8[1m]" }

in ~/.claude/settings.json.

I found this because I finally just realized there has to be something seriously wrong, I extracted every conversation I have had with claude for the last 2 years and did my own similar (not as well done but in depth) benchmarks and noticed a dramatic drop in performance. Basically useless.

Thank you.