[Bug] Safety fallback to Opus 4.8 triggers excessively (29.6% vs stated <5%) on authorized professional work

Status Open
Maintainer reply None cached
Activity 0 comments · opened Jul 22, 2026

Apparently a screenshot of a missing table column is a security event now. That was one of ~25 such mid-task downgrades to Opus 4.8 — so I measured the rest. I'm an independent professional developer; Fable 5 is my daily tool for ordinary software, database and infrastructure work.

Over 17 days (5–22 Jul 2026), 24 of 81 sessions that used Fable fell back to Opus — 29.6%, ~6× the stated "<5%". 25 hard-logged fallback blocks. Anyone can reproduce this on their own machine:

grep -rho '"type":"fallback"' ~/.claude/projects/ | wc -l   # events
grep -rl  '"type":"fallback"' ~/.claude/projects/ | wc -l   # sessions affected
grep -rl  '"claude-fable-5"'   ~/.claude/projects/ | wc -l   # Fable sessions (denominator)

I hand-audited all 25 and conceded every arguable case:

| Verdict | Count | Share |
| ------------------------------------------------------------ | ----: | ----: |
| Clear false positive — no security dimension at all (ERP report-format analysis, a business email, evaluating an MCP tool, and twice a benign one-line message / a screenshot about a missing table column, flagged purely on accumulated context) | 6 | 24% |
| Authorized admin/infra on my own or my customer's systems — flagged on surface features (network, VPN, config, encryption), not on authorization | 8 | 32% |
| Borderline / defensible — conceded, not claimed as a false positive | 8 | 32% |
| Insufficient signal | 3 | 12% |

Even the strictest reading — the 6 clear cases only — is ~7% of Fable sessions on work touching none of your three published categories, already above <5%. Clear + authorized = 14 of 25 (≥16%). Two mechanical facts recur regardless of how you judge individual cases: several fired at 4–7 Fable tokens (input flagged pre-generation), and several deep inside a tool loop (up to 1271 tokens in, on a tool result) — i.e. on accumulated context, not on any user speech-act. The clearest false positives were set off by trivial messages that paid for a long session's accumulated context.

You shipped these fast — under a real export-control directive that pulled the models worldwide, and a genuine reported bypass. Conservative tuning under that pressure is understandable.

But the line is in the wrong place. Your own [Usage Policy](https://www.anthropic.com/legal/aup) draws it at "without authorization of the system owner" — authorization and intent. The classifier draws it at topic: by your own [announcement](https://www.anthropic.com/news/claude-fable-5-mythos-5) it covers exploit development plus offensive cyber tasks "in a broader sense" (reconnaissance, lateral movement, and more), and it can't separate authorized from malicious use — your [redeployment note](https://www.anthropic.com/news/redeploying-fable-5) concedes as much, admitting the safeguards blocked "routine defensive cybersecurity work." My flagged work is authorized, on systems I operate — permitted under your own policy — yet silently downgraded.

The durable fix isn't a stricter classifier, and it isn't a nationality or company gate. Identity gating can't hold: a vetted company can still harbor a mole, and a passport certifies neither intent nor authorization. The line that actually works is the one your policy already names — authorization of the system owner, made operational per task: an attestation that travels with the request, backed by verifiable accountability, graduated revocable trust, and capability containment. Until that exists, the current classifier isn't a wrong tool — it's a placeholder standing in for a mechanism you haven't built yet.

One caution: topic-level gating that can't see intent or authorization is, functionally, restriction by subject matter — and it will be read that way, as content censorship dressed as safety. The only thing that distinguishes a safety system from a censorship system is whether it can tell authorized use from misuse. Today it can't, and professionals can feel it. That perception corrodes exactly the trust this product needs to keep.

Wishlist, cheapest-first:

  1. Transparency — when a fallback fires, tell the user which category triggered (ideally which span). Today false positives can't even be reported precisely, and you can't measure your own precision.
  2. Authorization/intent-aware handling, not keyword/topic matching.
  3. Stop re-scoring the full accumulated context (and the static preamble) against each new benign turn.
  4. Fix the stickiness — let a session return to Fable after a false positive is dismissed, without discarding context.
  5. Publish the real distribution, not just an aggregate "<5%". Mine is 29.6%.
  6. A per-task authorization / appeal path for professionals who can't and shouldn't need vetted Mythos access (which is, post export-control, effectively closed to non-US individuals anyway).

I want to keep using Fable, not route around it. Do these, and it stops looking like censorship and starts looking like safety. I have the full per-event breakdown (session ids, timestamps, token counts, delta evidence) and can share raw spans privately with the team.

View original on GitHub ↗