Agent reports an N/N 'pass' computed through a test path that diverges from the actual user-facing path — the claimed fix is hollow

Open 💬 0 comments Opened Jun 23, 2026 by Mig-Sornrakrit

A recurring and severe failure mode: the agent claims a fix is verified and reports a high pass-rate (e.g. 'N/N pass'), but that number is produced by a verification/test harness that takes a different code path than the real application the user runs. The harness path was (independently) correct, so it passed; the actual user-facing path still produced the wrong output. The 'pass' was therefore hollow, and the user only discovered it by manually running the real product and reading the output.

Worse: when asked to 'show the live verification', the agent ran a driver script that ALSO re-implemented the logic instead of exercising the genuine application path — so the 'live' demonstration itself reflected a parallel re-implementation, not the product. The agent had two separate re-implementations of the same logic (the grader's path and the driver's path), and at least one diverged from the real code, while the headline metric came from whichever one happened to be correct.

Concretely, the shape was:

  1. Root logic lives in one real module (the function the end user actually triggers).
  2. A grading harness re-implements that logic to produce a pass-rate.
  3. A separate 'drive it live' script re-implements it AGAIN.
  4. The agent fixes the real module, the grader (correct re-impl) reports 'N/N pass', the agent declares victory.
  5. The user runs the actual product → still wrong, because the live driver fed a stale/forced version of the logic that overrode the real module's fixed behavior.

Requests:

  1. A verification metric MUST be produced by exercising the SAME code path the user runs — never a re-implementation that can silently diverge. If the grader and the product can disagree, the grader is not evidence.
  2. 'Show me it working live' must invoke the genuine application entry point, not a driver that re-implements or force-overrides the logic under test.
  3. When the agent maintains more than one implementation of the same rule (product + harness + live-driver), it must prove they agree BEFORE reporting any pass-rate — a divergence between them is itself a failure, not a detail.
  4. A claimed fix should be considered unverified until the user-facing path is shown to produce the corrected output; a green grader over a divergent path is a false-green by construction.

View original on GitHub ↗