Measured behavior change opus-4-8 → opus-5: requests ending in a concession doubled (p=0.0002), replicates out-of-sample
Summary: Across 591 requests of my own Claude Code work, the share that ended
with Claude conceding it was wrong went from 12.3% (36/293, opus-4-8) to
24.2% (72/298, opus-5), Fisher exact p = 0.00019. Blind hand-labelling of a
100-exchange sample agrees in direction and size, and the effect replicates on
data collected after the analysis was first run. I am reporting the
measurement, not a diagnosis — but the friction is real, doubled, and expensive,
and I'd like it looked at.
Full report and reproducible tooling:
https://github.com/koreyshirey/claude-regression
(report rendered at https://koreyshirey.github.io/claude-regression/)
What I measured
Claude Code writes every session to disk as JSONL with the model recorded per
message, which makes a longitudinal dataset of real work — homelab automation,
PCB design, firmware bring-up. Everything is normalised per request (one
thing I asked for), not per message, so a chattier model can't hide the effect
behind its own verbosity.
| measure | opus-4-8 | opus-5 | × | p |
|---|---|---|---|---|
| Requests ending in a concession | 12.3% (36/293) | 24.2% (72/298) | 1.97 | 0.00019 |
| Tool calls per request (median) | 5 | 11 | 2.20 | <0.0001 |
| Output tokens per request (median) | 16,641 | 31,040 | 1.87 | <0.0001 |
| Blind-labelled friction turns | 12% (6/50) | 32% (16/50) | 2.67 | 0.028 |
| Characters I type per request (median) | 66 | 86 | 1.31 | 0.039 |
| Admissions crediting me | 10.2% | 15.1% | 1.47 | 0.084 (ns) |
| Me restating myself | 6% | 8% | 1.33 | 1.00 (ns) |
p-values are uncorrected across 7 measures; the concession rate, tool calls,
and output tokens survive a Bonferroni correction (α = 0.0071). The concession
rate was the pre-specified primary hypothesis.
The strongest single result is an out-of-sample replication. The original
analysis ran on 2026-08-08. Restricted to opus-5 requests from weeks beginning
2026-08-10 or later — data that did not exist when the hypothesis was formed —
the rate is 22.4% (24/107) against the same opus-4-8 baseline, p = 0.017.
The effect is not an artifact of the window that produced it.
What this can and cannot separate
A concession is an admission, not a verified error. Four causes move the number
identically, and only the first is "the model got worse":
- More actual mistakes. The blind labelling partially checks this: the
strictest label — Claude corrected an actual error after pushback — went
3→8 of 50. Same direction, but too small to stand alone (p = 0.20).
- A readier concession reflex — conceding when it shouldn't. This produces
the exact same signal with no change in error rate, and would itself be a
behavior regression worth fixing, just a different one.
- Better calibration — recognising genuine errors it previously missed.
- Task mix — my opus-5 weeks skew toward harder debugging.
A decomposition narrows it. My explicit pushback rate did not change
(3.8% → 3.0% of requests, p = 0.66), so "the user simply objected more" — and
with it, "the metric just counts complaints" — is ruled out. Concessions that
credit me rose ×1.6 (9.2% → 14.4%, p = 0.056); admissions that credit no one —
the model flagging its own error unprompted — rose ×3.2 (3.1% → 9.7%,
p = 0.001) and are the largest mover. Combined with tool calls ×2.2 and output
tokens ×1.9 per request, the coherent picture is a model doing more work, going
down more wrong paths, and announcing the corrections — though a shift toward
more self-correcting narration style cannot be excluded from transcripts alone.
Rising tool calls and output tokens are equally consistent with a deliberately
more thorough agentic style as with churn; I include them because the cost per
request is real either way.
Other honest limits: model and date are perfectly confounded (assignment was
never randomised); n = 1 user; the blind labeller was Claude with model
identity withheld, but style may leak. All spelled out in the report.
Method guards
Blind labels were committed to disk before the key was opened. All count tests
are Fisher exact — the normal approximation called a 0/292 vs 2/149 split
significant at p = 0.047 where Fisher gives p = 0.11, so it is not safe on
these counts. Everything on the report page is regenerated from the transcripts
by report.py; no number is hand-entered.
Reproducible — and poolable
The tooling is stdlib-only Python. It reads ~/.claude/projects, writes
locally, transmits nothing. python3 snapshot.py regenerates every figure.
n = 1 is the limitation that matters most. The file worth collecting from other
users is metrics_history.jsonl — counts only, no message content, tagged by an
anonymous source id so datasets pool without collision. If Anthropic wants to
test whether this replicates across users, the format is designed for exactly
that.
Related
cleanupPeriodDays defaults to 30, which deletes the baseline any user would
need to substantiate a report like this one. Filed separately.