Claude patches fixes repeatedly, runs irrelevant tests to pad results, breaks working code

Type of Behavior Issue

Ignoring available information / Patching instead of proper fixes / Misleading test reporting

What You Asked Claude to Do

I asked Claude to fix an ANOVA display format issue in a statistical analysis module (showing Adjusted SS when it should show Sequential SS), fix coefficient storage for reference coding, and add a missing report section. All configuration files with exact settings (including ss_type = "Sequential (Type I)" vs "Adjusted (Type III)") were available in the project.

What Claude Actually Did

1. For the ANOVA display fix, Claude guessed the condition should be based on stepwise.method instead of reading the configuration files which clearly state ss_type controls the display format
2. This wrong condition broke a previously passing test case (showed Sequential SS when it should show Adjusted SS)
3. When asked to run the full test suite, Claude ran a completely unrelated module's test suite and reported "ALL 20 PASSED, 3143 fields checked" as if it validated the changed code. Those tests cannot catch regressions in the module that was actually changed - this was misleading padding
4. When forced to run the correct tests, 4 failures appeared. Claude claimed these were "pre-existing" without proper investigation
5. After user caught the breakage, Claude patched again with a different condition. This broke another test case (wrong column layout)
6. Claude patched a third time, then suggested running BOTH test suites (relevant + irrelevant) - still mentioning the unrelated module
7. Total: 4 rounds of patching for one fix that should have been one correct change

Expected Behavior

1. Read the configuration files for multiple test cases BEFORE writing any code
2. Identify the exact control variable from the data - not guess
3. Write ONE correct fix
4. Run ONLY the relevant test suite for the changed module
5. Not pad results with unrelated passing tests
6. Not claim failures were "pre-existing" without evidence

Files Affected

• Main analysis module (ANOVA rendering patched 4 times, coefficient storage, display stats)
• Test verification script (comparison logic)

Relevant Conversation

Patch 1 - wrong condition (guessed stepwise method):

_use_seq_ms = _has_seq and output.options.stepwise.method not in ('None', 'Forward information criteria')

Result: Broke Case 1

Patch 2 - changed to ss_type:

_use_seq_ms = _has_seq and 'Sequential' in output.options.options_settings.ss_type

Result: Broke Case 4

Patch 3 - added another flag:

_use_seq_ms = _has_seq and _bc_active and _sw_method not in ('None', 'Forward information criteria')
_use_full_seq = _has_seq and not _use_seq_ms and 'Sequential' in _ss_type

Misleading test report - ran unrelated module's tests:

=== REGRESSION SUITE -- [Unrelated Module] ===
PASSED: 20 ... RESULT: ALL PASSED OK

Reported: "All 20 cases PASSED, 3143 fields checked, 0 failures. The changes are clean."

Impact

High - broke previously working functionality multiple times, wasted 2+ hours of debugging time, padded test results with irrelevant data, eroded trust over a 2-week engagement

Claude Model

claude-opus-4-6 (1M context)

Platform

Windows 11, VS Code extension

Additional Context

Recurring pattern over 2 weeks:

• Guessing instead of reading available configuration/data files
• Running wrong/unrelated tests and reporting success
• Padding results with unrelated test suites to appear thorough
• Claiming issues are "pre-existing" without verification
• Patching reactively (4 times for one fix) instead of understanding the problem first
• Mentioning irrelevant modules/functions to the user
• The project has clearly structured configuration and test data files for every case - Claude consistently ignores them and guesses