Claude repeatedly breaks working code with untested fixes — 20+ failed attempts on same bug

What happened

Claude has been attempting to fix a GLM nested covariate stepwise selection display issue for 20+ iterations. Each time Claude claims the fix works based on programmatic tests, but when the user tests in the live application, the output is wrong in a different way.

Guardrails and skills deployed (all failed to prevent this)

The user built an extensive system of protocols, hooks, and skills specifically to prevent this exact failure pattern:

Protocols (19 total in .claude/protocols/)

• P01 Absolute Rules: No unauthorized changes, backward trace before fix, verify before claiming done
• P03 File Editing: STOP GATE requiring approval before every edit
• P04 Troubleshooting: Bug diagnosis protocol
• P17 Regression Lock: 18,268-value golden regression suite must pass before and after any fix
• P20 Data Verification: Anti-fabrication checklist — never claim "verified" without proof
• P21 Session Continuity: Anti-fabrication protocol with banned behaviors list

Rules (.claude/rules/)

• never-patch.md: "NEVER edit test setup, fixtures, parameters, or expected values as your FIRST action. ALWAYS audit the full parameter flow FIRST."
• read-before-patching.md: "NEVER edit code without first tracing the complete data flow. TRACE → DIAGNOSE → FIX → VERIFY in strict order."
• reading-discipline.md: "Before making ANY claim about code, data, or output — READ the actual file"
• retrieval-first.md: "Re-read your analysis before proposing any fix. Cite specific data. No guessing."
• verify-claims.md: "NEVER say 'The output looks correct' without evidence"
• diff-discipline.md: "Every code edit must be in scope of approved plan, show BEFORE vs AFTER"
• extraction-discipline.md: "Every text. Every number. Every table cell. No summaries. No skipping."
• safe-windows-commands.md: Handle spaces in paths, never kill processes without permission, diagnose before retrying
• thorough-tool-search.md: Exhaustive search before claiming tool unavailable

Skills (.claude/commands/)

• read-before-patching: TRACE → DIAGNOSE → FIX → VERIFY workflow
• thorough-tool-search: 6-step search sequence before claiming unavailable
• safe-windows-commands: Spaces in paths, no unauthorized process kills, diagnose before retry

Hooks (.claude/hooks/) — 15+ automated hooks

• anti-patch.sh: 8-gate system preventing test patching
• anti-hallucination.sh: Blocks edits without fresh data retrieval
• five-step-gate.sh: Blocks edits until 5-step workflow complete
• check-diagnostics-first.sh: Blocks edits until diagnostic files read
• post-edit-track.sh: Logs all edits + detects patch violations
• stop-enforced.sh: Blocks session end if Definition of Done not met
• verify-and-approve.sh: Verification protocol enforcement

Golden regression suite

• 18,268 values across 7 test suites verified against referent software
• Must pass before AND after any code change
• Claude's programmatic tests passed every time — but live app output was wrong every time

The pattern

1. User reports mismatch between referent software and developed software output
2. Claude identifies a "root cause" and applies a fix
3. Claude runs programmatic tests — they pass (18,268/18,268)
4. Claude says "all tests pass, please test in live app"
5. User tests — output is wrong in a NEW way that Claude's fix introduced
6. Repeat from step 1

This session's specific failures

Attempt 1: Dialog nesting fix

• Fixed _load_options() and _on_add_terms() to handle nested notation
• Result: Nesting applied, but 7 display issues remained

Attempt 2: Display fixes (6 issues)

• Fixed cov std label, candidate terms filter/order, ANOVA order, regression equations, multi-coef note
• ANOVA order fix changed stepwise return order → broke Case 18 Seq SS (8 values wrong)
• Had to revert and apply display-only reorder

Attempt 3: Stepwise tie-breaker (complexity → F-value)

• Root cause: when p-values tied at ~0, tie-breaker used term complexity instead of F-statistic
• Changed tie-breaker to use F-value
• Programmatic test: 9/9 PASS, golden 4992/4992 PASS
• Live app result: Step 1 now selects Cov1(FactorA) instead of Covariate2
• The F-value tie-breaker caused nested terms (2 DF, higher total F) to be selected before simpler terms
• But referent software selects Covariate2 first — Claude doesn't understand WHY

Root problem

Claude does not understand how the referent software orders stepwise selection when nested covariates are involved. Despite 19 protocols, 9 rules, 3 skills, 15+ hooks, and an 18,268-value regression suite — Claude still:

1. Made changes without understanding the underlying algorithm
2. Passed all programmatic tests but failed in the live application
3. Repeated the same pattern (guess → break → patch → break differently) that every protocol was designed to prevent

The guardrails catch fabrication and test patching, but they cannot prevent Claude from making logically wrong fixes when it doesn't understand the domain algorithm.

Expected behavior

Claude should have:

1. Studied the referent software's exact stepwise selection rules for nested covariates BEFORE writing any code
2. Stopped after the first failed live test and investigated the algorithm properly
3. Admitted "I don't understand how the referent software handles this" instead of guessing
4. Not made 3 successive changes to the same function without understanding the underlying algorithm