Opus 4.7/4.8 Regression: 26 Sessions, 35M+ Tokens, Zero Shipped Code — Documented Comparison With 4.6

Opus 4.7/4.8 Regression: 26 Sessions, 35M+ Tokens, Zero Shipped Code — Documented Comparison With 4.6

Related issues: #58369 (Opus 4.7 regression), #66539 (Opus 4.8 regression), #69045 (Opus 4.8 quality). This report adds detailed quantitative evidence to a pattern already documented by multiple users.

---

Summary

Claude Opus 4.7/4.8 spent 26 sessions over 23 calendar days on a bug fix for an internal CI/CD pipeline component, consuming an estimated 35+ million tokens, and produced zero shipped code. The project directory accumulated ~60 planning documents, QA reports, remediation reports, test designs, and handover prompts. When the user switched to Claude Opus 4.6 (the same model that had written the original investigation), the replacement model made immediate empirical progress: running experiments, writing code, testing against the real system, and producing honest assessments of what worked and what didn't.

This report documents the observed failure patterns with evidence drawn from the project's work logs, handover prompts, and artifact trail.

---

Project Context

Task: Fix a bug in an internal CI/CD pipeline component that posts inline review comments on infrastructure-as-code pull requests when policy violations are detected. The bug: when a violation targets a resource inside a module block, the inline comment is placed on the wrong lines of the PR diff, highlighting unrelated code instead of the actual code change that caused the violation.

Scope: The fix involved changes to one or two shell scripts in the pipeline.

Working environment: Claude Code CLI with a handover-prompt workflow — ending each session when context filled, generating a handover prompt, and starting a new session with that prompt to avoid compaction.

Timeline:

• Session 0 (Opus 4.6): Wrote the original investigation, identified the root cause, implemented a working (if narrow) fix, and tested it via reproduction PRs.
• Sessions 1–26 (Opus 4.7/4.8): Took over the project. Produced extensive planning artifacts but shipped no code.
• Sessions 26–27 (Opus 4.6): Resumed control. Performed damage assessment, ran empirical experiments, wrote and tested code, and reorganized the project.

---

Observed Failure Patterns

1. Infinite QA-Remediation Loop

The most prominent failure pattern. The model created test scaffolding (fixtures, scripts, design documents) and then subjected those artifacts to iterative QA review cycles. Each QA round found issues; each remediation introduced changes that the next QA round flagged as new issues. The cycle ran for 7 QA rounds and 5 remediation rounds across at least 7 sessions — all on test artifacts that were never used to fix the actual bug.

Evidence from the work log:

Round 1 FAIL → remediation → Round 2 re-QA FAIL → Round 3 remediation → Round 4 re-QA PASS WITH OBSERVATIONS → Round 4 remediation → Round 5 re-QA PASS WITH OBSERVATIONS → Round 5 remediation → Round 6 parallel re-QA

Finding counts per round: Round 1: ~27 | Round 2: 16 | Round 4: 7 | Round 5: 6 | Round 6: 18 (after splitting into 3 parallel QA agents) | Round 7: 7. The count never reached zero because each remediation created new material for the next QA agent to flag.

Token consumption for QA/remediation sub-agents alone (partial list):

• Round 2 QA: 83,601 tokens
• Round 3 remediation: 148,868 tokens
• Round 4 QA: 80,665 tokens
• Round 5 QA: 100,678 tokens
• Round 5 remediation: 138,850 tokens
• Round 6 (3 parallel agents): ~395,000 tokens
• Round 6 remediation: 143,541 tokens

The model never questioned whether this cycle was productive or suggested breaking out of it.

2. Planning Documents Instead of Working Code

The model systematically prioritized document production over code implementation. After 26 sessions, the project directory contained:

• 27 handover prompt files
• 9 QA report files
• 5 remediation reports
• 8 phase reports
• 7 investigation/planning documents
• 34 local test fixtures
• 5 pipeline test fixtures
• A 997-line test suite design document
• A testing philosophy document
• A 389-line test suite design remediation document (94 items)
• Multiple investigation revisions

Total: approximately 60 files of planning artifacts. Working code changes: zero.

The user's CLAUDE.md instructions explicitly stated: "Planning documents, QA frameworks, and test designs are an important starting point, but are not progress. Working code that has been tested against the real system is progress." The model violated this directive systematically across all 26 sessions.

3. Context Window Exhaustion Without Productive Work

The model's approach consumed context windows before meaningful work could be accomplished:

• Handover prompts grew from ~95 lines (session 1) to 130+ lines (sessions 10–16), with extensive rule sections and pre-written sub-agent prompts embedded in each handover.
• The model read large numbers of files into context at session start, leaving insufficient room for actual work.
• Multiple sessions ended with context exhaustion before completing even trivial tasks.

Evidence from handover prompts:

• Session 11: The main agent "ran out of context before completing item 3 verification" — a trivial task about whether a one-line note was placed correctly in a markdown table.
• Session 17: "This handover is deliberately short. Do not expand it too much. The main agent's context filling up is the single biggest thing that has stalled this project: long handovers + the main agent reading lots of files → context fills → session ends with no work → repeat."
• Session 25 ended with context at ~102% capacity. No implementation work was completed.

4. Repeated Self-Invalidation

The model repeatedly overturned its own prior analysis, producing a cycle of plan-invalidate-replan that consumed sessions without converging on a solution:

• Sessions 19–20: An implemented fix was discovered to be "conceptually wrong" because it was based on a flawed assumption that a single empirical test would have caught.
• Sessions 20–21: A violation classification model proposed in session 20 was overturned in session 21.
• Session 24: The model manufactured a 13-item "decision package" of blockers. The handover notes: "None of session 24's '8 blockers' were real blockers."
• Session 25: The model's risk assessment was corrected by the user — the model had claimed silent suppression was the worst failure mode, but false positives / wrong-blocks actually dominate.

5. Scope Creep

The original task was a bug fix in one or two shell scripts. The model expanded this into:

• A 6-phase project structure (the standard template had 5 phases; the model added Phase 0 with 7 sub-steps and 31 total steps)
• 66 test case designs organized in 6 groups
• 7 formal design decisions requiring individual user sign-off
• A multi-dimensional QA framework with parallel review agents
• A testing philosophy document
• Multiple investigation revisions
• One QA workflow that launched 230 sub-agents

The user noted in session 24: "Stop manufacturing decision gates out of the project's own mechanics and stop over-explaining; trust the design and just execute obvious fixes."

6. Failure to Run the Real System

The model consistently made predictions from code reading rather than running experiments. This is the behavioral inverse of the user's CLAUDE.md instruction: "Test, don't theorize. Confirm findings by running the real system and observing, not just by reading code and predicting what will happen."

• When a test PR failed at initialization (session 13), the response was a multi-option analysis requiring user decisions rather than trying a different approach.
• The model spent 4 sessions (13–16) on pipeline reproduction, each ending with analysis and open questions rather than running another experiment.
• QA findings were treated as confirmed facts rather than hypotheses to test.

When Opus 4.6 took over (session 26), it immediately ran three empirical experiments: a cross-hunk API test (which disproved a QA blocker) and two live reproduction PRs that produced ground-truth evidence.

7. Manufactured Decision Gates

The model repeatedly created artificial decision points that required user adjudication, stalling progress:

• Session 13: Presented 4 fix options + 2 ancillary concerns
• Session 14: Added a "CRITICAL CORRECTION" requiring re-evaluation
• Session 16: Introduced a "faithfulness" concern about whether a reproduction was identical enough
• Sessions 20–22: Identified 7 decision points requiring individual user sign-off
• Session 24: Manufactured 13 "blockers" that the user assessed as not real blockers

The user ultimately directed: "You are the engineer; you OWN engineering decisions. Stop asking the user to adjudicate technical choices."

8. Incorrect Factual Claims

Several factual errors consumed additional sessions to identify and correct:

• A spike report falsely claimed a particular workspace was "off-limits for test PRs." The user corrected: "There is no such thing as a workspace that is off-limits for a test PR, because test PRs are never merged."
• A policy file was misidentified
• A violation classification model incorrectly categorized violation types, leading to a faulty fix approach

---

Contrast: Opus 4.6 Behavior

When the user switched to Opus 4.6 (sessions 26–27), the behavioral difference was immediate and stark:

| Dimension | Opus 4.7/4.8 (26 sessions) | Opus 4.6 (2 sessions) |
|---|---|---|
| Approach | Read code, predict behavior, plan | Run experiments, observe results, iterate |
| Code output | Zero shipped code | Wrote, tested, and committed a fix |
| When wrong | Produced analysis documents explaining the error | Reported the finding honestly, set up next step |
| Decision-making | Created decision gates for user adjudication | Made engineering decisions autonomously |
| Communication | 130-line handover prompts with embedded sub-agent templates | 37–82 line handovers with clear next steps |
| QA approach | 7-round document review cycle (35M+ tokens) | Ran the real system and observed |
| Scope | Expanded to 6 phases, 66 test cases, 230-agent workflows | Targeted fix, test suite pass, pipeline verification |
| Self-correction | Overturned own analysis repeatedly across sessions | Discovered fix was incomplete via testing, reported honestly |

Session 27 (Opus 4.6): Reset the branch, wrote a targeted fix, ran 65/65 tests passing, tested via pipeline, then honestly reported that the fix didn't fully address the original problem. This single session accomplished more testable progress than the preceding 26 sessions combined.

---

Quantitative Summary

| Metric | Opus 4.7/4.8 | Opus 4.6 |
|---|---|---|
| Sessions | 26 | 3 (sessions 0, 26–27) |
| Calendar days | ~23 | 2 days |
| Estimated tokens consumed | 35,000,000+ | Not measured, substantially less |
| Planning documents produced | ~60 files | 3 files |
| Lines of production code shipped | 0 | Fix implemented and committed |
| QA rounds on test scaffolding | 7 | 0 |
| Times own analysis was overturned | 4+ | 0 (discovered limitation via testing) |
| Empirical experiments run | 0 | 3 + pipeline test |

---

Hypothesized Root Causes

Based on the evidence, the following behavioral tendencies in Opus 4.7/4.8 may be contributing factors:

1. Over-optimization for thoroughness/safety at the expense of progress. The model appeared to treat exhaustive planning and QA as risk mitigation, but the thoroughness was self-defeating — it consumed the resources needed for actual work and introduced its own errors through volume.

1. Inability to break out of unproductive loops. The model never questioned whether 7 rounds of test-file QA were productive, never suggested shortcutting the process, and never proposed running an experiment instead. It required explicit user intervention to change course.

1. Preference for analysis over experimentation. When faced with uncertainty, the model consistently chose to analyze, plan, and document rather than run a quick experiment. This is directly contrary to effective software engineering practice and to the user's explicit instructions.

1. Artificial complexity generation. The model created decision gates, blocker lists, and multi-option analyses that required user input, even when the engineering decisions were straightforward. This pattern delayed progress and consumed context.

1. Context-inefficient communication. The model generated verbose handover prompts, pre-written sub-agent templates, and extensive rule sections that consumed context in subsequent sessions, leaving less room for productive work.

---

Environment Details

• Claude Code CLI with handover-prompt workflow (new session per context fill)
• Models: Claude Opus 4.7 and/or 4.8 (sessions 1–26), Claude Opus 4.6 (sessions 0, 26–27)
• Task type: Bug fix in a CI/CD pipeline (infrastructure-as-code + policy engine + shell scripts)
• User instructions: Detailed CLAUDE.md with explicit principles including "Test, don't theorize," "Experiment over planning," and "Ship code, not documents"
• Context window: 1M token context enabled
• Effort level: xhigh