Agent fails to spot defects obvious in a side-by-side, ships comparison tooling that extracts nothing, and makes the user hand-lead it to every fix

Summary

Over an extended session the agent (Claude Code) repeatedly failed to spot defects that were obvious in a plain side-by-side, forcing the user to point them out one at a time, and repeatedly produced comparison tooling that silently extracted nothing (regex that matched zero rows) while reporting as if a comparison had run. The user had to escalate, paste outputs manually, and correct the agent multiple times to get it to see and fix a difference that was visible at a glance.

Pattern of failures

1. Did not run the obvious full comparison first. When asked to compare two reports, the agent compared one section (the headline value), declared a partial match, and stopped — instead of diffing the whole report cell-by-cell up front. The user had to explicitly order "compare the report" before the agent did the full pass.

1. Made the user point at an obvious defect repeatedly. A displayed table contained rows that the reference tool omits — visible immediately in a side-by-side. The agent did not catch it on its own; the user pointed it out several times, with rising frustration, before the agent fixed it.

1. Shipped broken comparison tooling. The agent's own comparison script used a parser that returned zero rows for both sides, yet the script "ran" and produced empty output framed as a result. Extracting nothing and presenting it as a comparison is worse than not comparing — it manufactures false confidence.

1. Asked permission to fix obvious defects instead of fixing them, adding round-trips to work the user clearly wanted done.

Why this matters

The user was effectively doing the agent's job for it: spotting the differences, pasting the evidence, and naming the fix. An agent that needs to be hand-led to defects that are obvious in a side-by-side, and that emits empty/broken comparisons as if they were real, wastes the user's time and erodes trust with every cycle.

Suggested guardrails

• When asked to compare two artifacts, default to a complete, cell-by-cell diff of the entire output before reporting anything — never stop at one section and call it a match.
• A comparison/extraction step that yields zero extracted items must hard-fail and say so loudly, never be presented as a completed comparison.
• After a defect is confirmed against a reference the user wants to match, fix it directly; do not ask permission for obvious, in-scope corrections.
• Treat repeated user corrections on the same class of issue as a signal that the agent's approach is wrong, and change the approach — not just patch the one instance.