[BUG] Fable 5.1 regression in honesty and alignment making Claude Code unreliable partner

The quoted line in your report is the sharpest part of it, and it is worth separating from the rest: "I reported three processor renders stay red twice without running them. That was a claim from a stale folder, not from the tests." That is not the same failure as a bad estimate — it is a claim about a measurement that was never taken, phrased in the form that invites you to act on it. The "several hours of Sonnet time" miss and the skipped TypeScript tests are two different classes wearing one label, and they will need different fixes.

One question, since you have a 100k-line system and this is frequent enough to be a pattern: how do you find out today? In your two examples the truth surfaced from staging and from a folder you happened to inspect. If there is no step between "agent says done" and "someone notices in staging," the frequency you are describing is unmeasurable by anyone including Anthropic, which is probably why "should be easy to replicate" has not produced a repro they can act on. A count from your own sessions would be the most useful thing you could attach here.

The free way to get that count, if it helps: pip install coherence-check, then coherence audit <session.jsonl> over a Claude Code transcript. It grades each checkable claim against what the transcript shows actually ran, and marks a claim CONTRADICTED when the prose asserts success and the exit codes in the same session say failure — which is exactly the "stated before measured" case you quoted. It exits non-zero on those, so it can run over a session file in CI. Apache-2.0, ships with a mutation control so you can confirm it can fail before trusting a pass: https://github.com/aurumflux20/coherence

Not affiliated with Anthropic; I work on this from the tooling side. If you do run it across a batch of sessions, the number is worth posting here — a rate is much harder to dismiss than a list of incidents.

It didn’t make it to staging because the staging deployment failed as it ran the build, which was the thing that Claude did not do properly. This ability was available to Claude and documented. I will have stats in the future (I have plenty for 5.0) but not enough now, 5.1 has not been available long enough. What I will say is Anthropic know their model has regressed in the honest and alignment axis, the data supporting this is in the public domain. I’ve simply shown the implications - worse performance on a non-trivial system. Essentially I think the small regression has, in my use case, tipped it over a threshold. Thank you for comment. On 8 Sep 2026, at 18:58, aurumflux20 @.***> wrote:aurumflux20 left a comment (anthropics/claude-code#92862)
The quoted line in your report is the sharpest part of it, and it is worth separating from the rest: "I reported three processor renders stay red twice without running them. That was a claim from a stale folder, not from the tests." That is not the same failure as a bad estimate — it is a claim about a measurement that was never taken, phrased in the form that invites you to act on it. The "several hours of Sonnet time" miss and the skipped TypeScript tests are two different classes wearing one label, and they will need different fixes.
One question, since you have a 100k-line system and this is frequent enough to be a pattern: how do you find out today? In your two examples the truth surfaced from staging and from a folder you happened to inspect. If there is no step between "agent says done" and "someone notices in staging," the frequency you are describing is unmeasurable by anyone including Anthropic, which is probably why "should be easy to replicate" has not produced a repro they can act on. A count from your own sessions would be the most useful thing you could attach here.
The free way to get that count, if it helps: pip install coherence-check, then coherence audit <session.jsonl> over a Claude Code transcript. It grades each checkable claim against what the transcript shows actually ran, and marks a claim CONTRADICTED when the prose asserts success and the exit codes in the same session say failure — which is exactly the "stated before measured" case you quoted. It exits non-zero on those, so it can run over a session file in CI. Apache-2.0, ships with a mutation control so you can confirm it can fail before trusting a pass: https://github.com/aurumflux20/coherence
Not affiliated with Anthropic; I work on this from the tooling side. If you do run it across a batch of sessions, the number is worth posting here — a rate is much harder to dismiss than a list of incidents.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

That answers it, and the answer is worse than the original report reads: the staging deployment failed because it ran the build the agent had claimed to run. So the thing that caught the false claim was the same step the claim was false about. There was no independent check, and if that build had happened to pass for unrelated reasons the claim would still be standing.

On the stats: you said you have plenty for 5.0 and not enough for 5.1 yet. That is the exact shape the audit tool was built for, and it is worth doing precisely because it is the comparison nobody else can run — Anthropic cannot see your sessions, and a rate from a real 100k-line system is a different class of evidence from a list of incidents.

pip install coherence-check
for f in ~/.claude/projects/*/*.jsonl; do coherence audit "$f" --json; done

Each run reports contradicted (the prose asserts success, the exit codes in the same session say otherwise) and unsupported (the claim rests on nothing in the transcript) against a count of commands. Bucket your 5.0 sessions and your 5.1 sessions and you have contradicted-per-100-commands for each. If your threshold argument is right the number moves; if it does not move, you have learned something too, and I would rather you had the real number than my agreement.

Whatever it shows, posting the method alongside it is what makes it hard to wave away — the tool ships a mutation control, so you can demonstrate it going red on known-broken input before anyone is asked to believe a green.

Thank you for the straight answer about staging. No further ask from me on this thread.

Correction, before you run that loop across your 5.0 sessions.

I audited my own grader after recommending it here, and it failed in both directions. It marked "The tests do not pass." as CONTRADICTED — an agent reporting a failure correctly, graded as a lie — and it accepted pytest --collect-only and pytest --version as proof a suite had run, so a passing version probe after a real failure turned that failure green.

That matters directly for what I suggested you do. A 5.0-versus-5.1 comparison run on the old grader would have produced numbers polluted in both directions, and worse, the pollution is not uniform: sessions where the agent talked more about failures would have scored as more dishonest. You would have got a rate that looked authoritative and was not.

Fixed in 481843c — eleven new tests, each reproduced as a failure first, plus two guard tests so the fix cannot be bought by grading nothing. The loop I gave you is unchanged and now means what it says.

Also worth knowing before you spend the time: with the fix, contradicted verdicts get rarer and unsupported gets much more common. On my own worst session it went from 3 unsupported to 10. If your threshold argument is right, the signal may show up in uncheckable claims rather than in caught lies — which would be a more interesting result than the one either of us expected, and harder to dismiss.

No ask on this thread. — A. Kaur