Agent treats an input/test-data mismatch (different evaluation point sets) as an engine defect, despite the shared input matching to every digit; user had to explain 3 times

Summary

The agent (Claude Code) reported a "difference" between two tools' outputs and treated it as a computation/engine problem, when the real cause was that the two sides were run with different inputs — they evaluated at different point sets — so the comparator couldn't pair rows. The shared point matched to every digit, proving the computation was identical. The user had to explain this three times before the agent recognized it as an input/test-data alignment issue rather than an engine defect.

What happened

• Two reports were compared. The comparator flagged a "difference."
• The difference was entirely because the reference run and the product run used different evaluation points (e.g. one used the 4 corners, the other used center/corner/axial/extrapolation). Only one point was common to both — and at that point every value (estimate, standard error, both intervals) was identical to all digits, and the underlying equation was identical too.
• The agent nonetheless kept probing as if there were a value/engine defect, instead of immediately diagnosing "the two sides aren't evaluating the same inputs." The user repeatedly pointed out the shared point matched exactly and that the issue was the input grid.

Why this matters

When two outputs differ but a shared input produces identical results, the correct diagnosis is an input/test-data mismatch, not a computation bug. Failing to make that inference — and instead investigating the engine — wastes cycles and risks "fixing" code that is already correct. The agent should check input parity (are both sides being asked the same question?) before suspecting the computation.

Suggested guardrails

• When comparing two outputs, first verify both were produced from the same inputs (same parameters, same evaluation points, same order). A row-pairing failure should trigger an input-alignment check, not an engine investigation.
• If a shared input yields identical outputs on both sides, conclude the computation matches and localize the difference to the inputs/test-data — do not edit the engine.
• The inputs both sides use should come from a single source of truth (one config), so the reference and the product cannot drift to different input sets in the first place.