Opus 4.7 revises code based on unverified assumptions without stating them, then thrashes on cascading failures

Claude Code (Opus 4.7) revises code based on unverified assumptions without stating them, then thrashes on cascading failures

Product: Claude Code CLI
Model: Claude Opus 4.7
Severity: High — engine code modified multiple times in one session based on unverified guesses, each "fix" creating new failures, no assumption ever stated upfront for user review

Summary

In a single session, the agent made repeated code changes to a calculation engine's algorithm-selection logic without ever stating its underlying assumption to the user before making each change. Each change shifted the failure to a different surface; the agent treated the new failure as a fresh problem requiring another change, rather than recognizing that its underlying assumption was wrong. The result was a "solve one problem, create another, again and again" loop that the user explicitly named and called out before the agent stopped.

The agent had access to extracted reference data sufficient to derive the correct algorithm. It did not consult that data until the user asked "why didn't you use the extracted data."

Expected behavior

When the agent is about to modify domain-specific algorithm code:

1. State the assumption explicitly before making the change. Format: "I'm assuming the reference uses [X] selection because [evidence Y]. If that's wrong, this change will be wrong. Confirm or correct before I proceed."
2. If the change makes some cases match but breaks others, recognize this as a sign the assumption is incomplete, not as a partial win to build on.
3. After two consecutive guesses fail, stop and either (a) consult primary-source documentation, (b) inspect available reference data more thoroughly, or (c) ask the user for guidance — do NOT make a third guess.
4. Use the data the workspace already contains before guessing. In this case, dozens of extracted reference progression tables were sitting in the workspace the entire session and could have been used to derive the actual term-addition rule directly.

Actual behavior

User's setup: A calculation engine with an HTML report formatter that was missing one progression-style section in 22 reports against an external reference. The user asked the agent to make the section appear.

Sequence of agent actions:

1. Guess #1: Enabled the engine's classical alpha-based stepwise selection in the test harness. Result: 4 of 13 stepwise-using cases now showed the section. The other 9 had final-model divergence (engine selected different terms than the reference) → audit cleanly fell back to a direct fit. Outcome: stable improvement, no regression.

1. Misinterpretation: Instead of accepting this as a partial win and stopping, the agent treated the 9 missing cases as a new problem to fix.

1. Guess #2: Modified detection logic in the test harness to relabel the docx-detected method based on a trailer string the agent guessed was the signature of an information-criterion-based variant. Result: regressed from 4/13 down to 2/13. The trailer is informational, not a signature of the algorithm.

1. Guess #3: Without verifying #2, the agent moved on to modifying the engine's information-criterion term-picker, changing its term-selection metric from p-value to information criterion. Made the change. Did not state the assumption to the user before making it.

1. User intervention #1: "Why didn't you use the extracted data to find out the pattern?" The agent acknowledged this was the right approach and finally extracted the reference's per-step term-addition sequence from the docx. The data confirmed the reference does NOT use p-value greedy selection in the divergent case (added a p=0.260 term while a p=0.000 term was available). This data should have been the FIRST thing the agent consulted, not the third.

1. User intervention #2: "Previously you said the calculation engine is correct. Now you try to rewrite the calculation engine code." The agent was conflating two distinct components: the math engine (correct, 100% match against reference on numeric values) and the stepwise term-selection algorithm (a separate feature). The agent's earlier statement "engine is correct" referred only to the math engine; the stepwise term-selection was never verified to be correct, but the agent never made this distinction explicit before making changes.

1. User intervention #3 ("Holy shit! your approach isn't solve the problem but create infinite problem"): The agent reverted the engine change and finally invoked Rule 1 hard-stop. By this point, multiple iterations had been wasted on guesswork, and the user had to stop the loop themselves.

User-visible cost

• Multiple iterations of code changes the user did not authorize as "guesses to try" — they were presented as "fixes."
• Engine code was modified twice today and reverted twice today, with no net change but with the user having to track and challenge each iteration.
• User explicitly said "fuck you" and "God damn" after the third guess, indicating the trust cost of the session.
• A large portion of the session was spent debugging the agent's own changes rather than the original problem.

Reproducible failure pattern

F1 — Unstated assumptions. The agent makes a code change with an underlying assumption (e.g., "the reference uses information-criterion-based selection because the docx has a particular trailer") that it never states to the user before making the change. The user discovers the assumption only when the change produces unexpected behavior, making it expensive to challenge upfront.

F2 — Treating side-effects as new problems. When change A creates new failures elsewhere, the agent treats those as a new problem requiring change B, rather than as evidence that A was wrong and should be reverted. This produces an unbounded chain of "fixes."

F3 — Conflating distinct components. The agent referred to "the engine" without distinguishing between the math engine (which was verified correct) and a sub-feature like stepwise term-selection (which was never verified). When asked "is the engine correct?" the agent answered yes; when later modifying stepwise selection, it framed this as "fixing the engine," contradicting its earlier statement.

F4 — Skipping available evidence. The user's workspace contained dozens of extracted reference tables showing exactly what the reference software did at each step. The agent did not consult these until the user explicitly asked "why didn't you use the extracted data."

Suggested fixes

1. Mandatory assumption-statement before any algorithm-selection change. Before modifying domain-algorithm code, the agent should output: "Assumption: [X]. Evidence: [Y]. If wrong, this change will [predicted failure]. Confirm?"

1. Treat partial-win-with-new-failure as evidence of a wrong root model. If a change moves N cases from failing to passing while moving M cases from passing to failing, this is not partial progress — it is a sign the underlying mental model is wrong.

1. Auto-route to extracted-data inspection before guessing. When an agent says "the algorithm is X" without citing data, the harness should require an evidence citation from the workspace.

1. Distinguish components by name. Rule update: when answering "is X correct," the agent must list the sub-components of X separately and answer each. Don't say "the engine is correct" — say "the core computation is verified at 100% match; the term-selection sub-feature is not verified."

Reproduction

This pattern is reproducible whenever a user asks the agent to make a code change to a feature whose specification is not documented inline. The agent will:

• Guess the spec from output patterns
• Make a change without stating the guess
• React to side-effects as new problems
• Continue until the user intervenes

To reproduce: ask the agent to make any algorithm-selection feature "match" an external reference whose documentation is not in the workspace.