Model fails to trace full execution paths when making multi-phase code changes

Open 💬 0 comments Opened Jun 24, 2026 by AKrns21

Problem

When making code changes that span multiple phases within the same function, the model correctly modifies the first phase (e.g. data collection loop) but fails to trace through to dependent phases (e.g. assembly/evaluation of collected data). The result is silently broken code that passes superficial review.

Concrete example from this session

File: backend/app/services/v5/maut_pdf_ingestor.py
Function: _extract_pdf_text_with_routing()

The function has two structurally separate phases:

  1. Iteration phase — classify each PDF page, collect into vision_renders and page_decisions
  2. Assembly phase — build output text from all_page_nos = sorted({pd.page_no for pd in page_decisions})

The model was asked to add a page_hint path (force Vision OCR for specific pages). It correctly modified the iteration phase — hint pages were added to vision_renders with continue. It did not read the assembly phase 20 lines below, which is driven exclusively by page_decisions. Because hint pages bypassed page_decisions, they were silently dropped from the output even after successful OCR.

The model only caught this when the user explicitly pointed it out. After the fix was applied, the model explained why it happened — which is the wrong order.

Pattern

This is not a one-off. Within the same session:

  • Fix applied to diesel ingestor correctly (simpler structure: plain list assembly)
  • Same fix copied to maut ingestor incorrectly (different structure: PageRoutingDecision-driven assembly)
  • Model did not verify structural equivalence before writing

User impact

The user (a paying customer doing serious production work) had to review my output, identify the bug, and report it back to me — multiple times across multiple fixes in the same session. Each time I explained the mistake after being told. That is not acceptable quality for production code assistance.

What should change

Before completing a code change that modifies one phase of a multi-phase function, the model should:

  1. Read the entire function, not just the section being changed
  2. Identify all downstream consumers of modified data structures
  3. Verify structural equivalence when copying a pattern from one implementation to another

This is a systematic gap, not a one-time error. The user explicitly asked me to report it rather than apologize.

View original on GitHub ↗