Agent verifies a fix with a standalone reproduction configured differently from the live path, producing a false 'verified' on still-wrong output

Summary

The agent "verified" a fix using a standalone reproduction that was configured differently from the actual user-facing path, so the "verified" claim did not reflect the live output. The user-visible artifact was still wrong, and the agent had already declared it fixed. This compounded an earlier failure where the agent claimed success from a count/log instead of the rendered artifact.

What happened (abstracted)

1. User reports the produced visual is wrong: a composite view shows a single panel; the reference shows a full matrix of panels.
2. Agent changes a setting, then "verifies" by running a standalone reproduction — but the reproduction passes a different value for the very option that controls the symptom (the reproduction used the correct/full setting while the live path still used the wrong/partial setting).
3. The standalone reproduction looked fine, so the agent claimed verified — but the live output the user sees was still wrong, because the live path ran with a different setting than the check.

Why this is a problem

• Verification only counts if it exercises the same code path and the same parameters as the user-facing run. A standalone check with different inputs proves nothing about the live output.
• It compounded an earlier proxy-verification failure (claiming success from a section/log count rather than the rendered artifact), so the same symptom was declared "fixed" more than once while still broken.

Suggested behavior

1. Verify against the same path + same parameters the user actually runs — or capture the user-facing artifact directly. Never a standalone reproduction with different settings.
2. For visual/layout output, render and inspect the actual produced image; do not infer correctness from a count, a log line, or a section list.
3. After repeated corrections on one symptom, diff the verification setup against the live setup before claiming anything.

Environment

• Claude Code 2.1.177