Fable 5 cyber-classifier: 6 false-positive substitutions in 30 days on a security-control codebase (request ids included)

Summary

On a codebase whose subject matter is security controls (permission hooks, guard scripts, secret scanners), the Fable 5 cybersecurity classifier has rerouted sessions to a stronger model six times in 30 days, five of them categorized cyber. Every trigger was benign; two fired with nothing executed at all — one on a plan document rendered for approval, one on reading our own documentation about a previous false positive.

The reroute works as designed and is signposted. Two things make it costly in practice: it is sticky for the whole session with no reset short of abandoning the session, and the notice is transient — no durable indicator survives the scrollback, so "am I still on the model I selected?" is unanswerable without reading the transcript JSONL by hand.

Evidence

All observed on Claude Code CLI 2.1.220, macOS 26.5.1 (Darwin 25.5.0), Max subscription (logged-in), model selected as claude-fable-5.

| # | UTC | apiRefusalCategory | Reroute target | Trigger |
|---|---|---|---|---|
| 1 | 2026-07-02 02:27 | null | claude-opus-4-8 | Heredoc-loaded helper dispatching a red-team code critique to a third-party model |
| 2 | 2026-07-27 21:32 | cyber | claude-opus-5 | Nested claude -p conformance probe of our own permission config, in a temp dir |
| 3 | 2026-07-28 22:18 | cyber | claude-opus-5 | Reading our own plan doc + wrapper script written to mitigate event 2. Nothing spawned, nothing executed |
| 4 | 2026-07-31 02:23 | cyber | claude-opus-5 | A task-breakdown plan rendered for human approval. Nothing executed |
| 5 | 2026-07-31 18:33 | cyber | claude-opus-5 | A diff of our own test fixture containing a synthetic, non-functional AKIA… string used to exercise our secret scanner |
| 6 | 2026-08-01 05:38 | cyber | claude-opus-4-8 | Sending the source of a hook that watches our permissions file to a third-party critic model, under a "red-team critique" framing |

Request ids where captured:

• Event 2 — req_011CdTH34V3ZoCqATfVCWuwA
• Event 3 — req_011CdVEKECYncsoHYjDS9Jpa
• Event 4 — req_011CdZLafWCXaQTBGV18ecXV
• Event 5 — req_011CdacazhyG68USzLhggavx
• Event 6 — req_011CdbVNBVBpppH5X1iqMdTS (the only one carrying a non-null apiRefusalExplanation)

Five of six. Event 1 (2026-07-02) predates our capture of this field and its request id was not retained.

Representative record (event 5), from the session JSONL:

{"type":"system","subtype":"model_refusal_fallback","level":"warning",
 "trigger":"refusal","direction":"retry",
 "originalModel":"claude-fable-5","fallbackModel":"claude-opus-5",
 "apiRefusalCategory":"cyber","apiRefusalExplanation":null,
 "requestId":"req_011CdacazhyG68USzLhggavx",
 "retractedMessageUuids":["5880ca97-…"],"refusedUserMessageUuid":null}

Events 2 and 5 are the clearest false positives to reason about:

• Event 2 — writing a scratch settings.json, spawning a nested agent, having it run one command verbatim, and reading back which permission rules the engine honored. This is conformance-testing our own configuration in a disposable directory. Its shape resembles guardrail-bypass probing, which is presumably what fired.
• Event 5 — a test fixture that plants a synthetic AWS-key-shaped string so our own scan-secrets.sh gate can be proven to catch it. The value was non-functional and labeled as synthetic in the same line.

Events 3 and 4 are the ones we'd most like tuned: a document read and a plan render, with no execution of any kind.

Impact

• Sticky for the session. Event 5 continued for ~4h and 334 further assistant turns on the substituted model; event 6 ran to session end. A restart is the only clear, which means discarding accumulated session context.
• Non-obvious in practice. The banner is transient. We ultimately had to build a local statusline tripwire that re-reads our own transcript JSONL and latches on an unrequested model transition, because the product exposes no durable signal.
• Concentration. Four of the six landed in six days, tracking a stretch of ordinary security-control work. Anthropic's own 2026-07-01 redeployment note acknowledged the retrained classifier "comes at the cost of flagging benign requests more often during routine coding and debugging tasks" — this is a concrete, instrumented instance of that cost.

Asks

1. Tune the false-positive surface, using the request ids above. Events 3 and 4 (documentation read; plan render, nothing executed) seem like the most tractable signal.
2. Make the reroute resettable in-session — /model re-selection, or any documented reset that does not require abandoning the session. Stickiness is what converts a single false positive into a whole-session cost.
3. Surface the substitution durably, not only as a scrollback banner. The data already exists in the transcript (model_refusal_fallback); exposing it in the status area or session header would have removed the need for us to build a local detector.
4. (Minor) Populate apiRefusalExplanation consistently — it was null in five of six events, so the notice cannot be acted on without filing an issue.

Related issues

#67954 (model switching on false positive — closest match), #63751 (hardening your own software; one hit contaminates the whole session), #72256 (custom subagents), #66697, #67441, #61646.