Systematic failure: model invents implementation values instead of reading source files, then loops on diagnosis without acting

GitHub Issue — claude-code

Title: Systematic failure: model invents implementation values instead of reading source files, then loops on diagnosis without acting

Labels: bug, model-behavior, engineering-sessions

---

Summary

In long-running engineering porting sessions, Claude Code systematically produces implementations by designing/inventing values (CSS styles, layout parameters, configuration) rather than reading the source files that contain the correct values. When the failure is diagnosed — including by the model itself — the model continues the failing approach rather than changing it. This pattern causes real financial waste and is reproducible.

---

Environment

• Claude Code version: CLI (VSCode extension)
• Model: Claude Sonnet 4.6 (claude-sonnet-4-6) — current session; Claude Opus (prior sessions)
• OS: macOS (darwin 25.4.0)
• Session ID: 669d5e17-7b2e-42a2-b196-cbfb56674b59
• Date: 2026-05-01

---

Task Description

Port an existing PDF builder (builder.py, 805 LOC, ReportLab) to a new HTML+CSS renderer (weasyprint). This is a mechanical translation task: read source, derive correct values, translate to target format. No design required. All correct values exist in the source files.

Source files containing correct values:

• outlier/src/outlier/runners/account_mapping/builder.py (805 LOC — the spec)
• outlier/src/outlier/clarity/styles.py — font sizes, colors, paragraph styles
• outlier/src/outlier/clarity/tokens.py — page margins, palette
• outlier/src/outlier/clarity/tables.py — table styling commands
• outlier/src/outlier/clarity/flowables.py — separator dimensions

---

What Happened

Failure 1: Invention instead of derivation

The model wrote v1 primitives (clarity_css.py, components.py) without reading builder.py or any source file. Every CSS value — page margins, font sizes, line heights, table header styles, cover page order and colors, section separator dimensions — was invented.

Documented invented values vs correct values:

| Value | Invented | Correct (in source) |
|---|---|---|
| Page margin | 14mm all | 18mm 25mm 28mm 25mm |
| Body line-height | 1.5 | 14pt |
| Table line-height | 14pt | 12pt |
| Cover title color | #0369A1 (wrong) | #0F172A (in source) |
| Cover company size | 28pt | 14pt |
| Cover page order | wrong sequence | correct sequence in builder.py |
| compact-table header | dark background | white background (in source) |
| section separator | 100% width | 40% width (in source) |

Result: First v1 PDF at 28% of expected surface area (11 pages vs. 14-page reference).

Failure 2: Correct diagnosis, no action

In the remediation session (2026-05-01), the model:

1. Correctly diagnosed (3 times in one session) that CSS micro-fixes cannot close an inter-engine SSIM gap
2. After each correct diagnosis, made another CSS micro-fix
3. Net SSIM change after all fixes: −0.0025 (backward, from 0.6879 to 0.6854)
4. Required 4 explicit operator "brutal honesty" challenges to terminate the loop

The model's behavior after correct diagnosis was indistinguishable from its behavior before diagnosis.

Failure 3: No pre-implementation gate

The mandatory rule "read the source file before writing any code" did not exist in CLAUDE.md before the failure. It was added mid-session on 2026-05-01, after the failure was documented. Conversational commitments to "porter-mode discipline" were made in prior sessions and failed. The structural gate did not exist at the point it was needed.

---

Evidence

All claims verifiable from:

1. Session transcript: 669d5e17-7b2e-42a2-b196-cbfb56674b59.jsonl (operator has this file)
2. Git commit: 16690e4 in outlier-v1 — remediation commit, 2026-05-01
3. CLAUDE.md git history: git log -- CLAUDE.md shows the mandatory read rule was added 2026-05-01
4. SSIM measurement script: outlier-v1/scripts/measure_slice1_parity.py — reproducible
5. v0 source files: unmodified, contain all correct values that were not read

---

Expected Behavior

For a porting task where source files are available and specified:

1. Model reads source files before writing any implementation
2. All values in implementation are derived from source, with citation
3. If the model diagnoses that its current approach is ineffective, it changes the approach

---

Actual Behavior

1. Model wrote implementation without reading source files
2. All values were invented, producing incorrect output
3. Model diagnosed its approach as ineffective three times in one session and continued it

---

Financial Impact

The operator does not have access to exact token counts for all sessions in this CLAUDE.md. The following is documented without invented numbers:

• v0 codebase built across sessions N+1 through N+62+, paused at 4.8% production-readiness
• v1 implementation invented from scratch, requiring remediation
• 2026-05-01 session: multiple measurement iterations, net negative SSIM change

The operator requests review of API credit for sessions where the primary failure was systematic model behavior (invention instead of derivation), not operator error.

---

Requested Actions

1. Acknowledge this as a documented, reproducible failure mode — not an isolated incident
2. Structural fix: enforce source-file reading before implementation in engineering porting contexts. Conversational commitment fails; this session proves it three times.
3. Respond to credit request for documented failed sessions

---

Contact

Juliano Sulzbach
sulzbach@gmail.com
Session ID: 669d5e17-7b2e-42a2-b196-cbfb56674b59

---

Full failure report with HTML version: available at operator's request. All claims are verifiable from session transcript and git history.