Agent analyzes a difference between two outputs without first checking they used the same setup — recurs daily despite hooks, memory, instructions, and mechanical locks

Summary

When asked to explain a discrepancy between a reference output and a candidate output, the
agent spent several turns analyzing why the two values differed — instead of first
verifying that both were produced from the same configuration. The discrepancy turned out
to be caused by the two sides being run with different setups, so the entire causal
analysis was moot. The project had an explicit standing instruction to verify setup-equality
before attributing a difference, the agent had read it earlier in the session, and it did not
apply it at the decision point.

What happened

1. A reference output and a candidate output disagreed on a value.
2. A standing instruction in the repo required: before treating a difference as a real

behavioral divergence, verify that both sides were run on the same case with the same
setup/configuration.

1. The agent instead:

• ran an independent tool to reproduce each side's number,
• built a table of intermediate quantities to characterize the gap,
• proposed which side was "right" and framed it as a genuine defect to fix.

1. Across multiple turns of user correction, each of the agent's replies produced more

analysis of the difference rather than stepping back to check whether the two runs were
even configured the same.

1. Only after the user explicitly said "verify they used the same setup" did the agent check

the actual configuration on each side — at which point the difference was a configuration
mismatch between the two runs, not a behavioral difference. The whole analysis was
irrelevant.

Why this is a defect

• When comparing two things, the first question must be **"were they given the same

inputs/configuration?"** — not "why do their outputs differ?". The agent inverted this.

• There was an explicit instruction to perform exactly this check first. The agent had read

it earlier in the same session but failed to apply it at the moment the comparison was made.

• The failure compounded turn over turn: instead of self-correcting to the setup check, each

turn doubled down on explaining the difference, until the user forced it.

Expected behavior

Before analyzing or attributing a discrepancy between two outputs, verify that both were
produced under identical configuration/inputs. A difference between two runs is not
evidence of a behavioral defect until the runs are confirmed to be the same setup. If a
standing instruction requires that check, apply it at the point the comparison is made — not
only after the user demands it.

Recurrence — and why the existing safeguards do not stop it

This is not an isolated incident. The same failure recurs daily, despite a substantial
enforcement layer already in place:

• automated pre-action hooks / gates (fail-closed checks that run before tool calls),
• persistent cross-session memory recording prior corrections,
• explicit written instructions / rules that mandate the precondition check,
• mechanical locks that block classes of unsafe edits.

None of them prevent this class of mistake. The reason: these safeguards enforce *syntactic /
file-level* conditions (which files may be edited, whether an artifact exists, whether a
message contains a banned word). The precondition that was violated here is semantic —
"before attributing a difference between two outputs, confirm both were produced from the same
configuration." No hook fires on that; it depends on the agent choosing to apply a rule it has
already read. So the agent reads the governing document at session start, then walks straight
past the check at the decision point where it binds, and the whole scaffolding — hooks, memory,
instructions, locks — is silent while the agent produces multiple turns of wrong analysis.

The core problem is that a rule the agent must remember to apply is not reliable, even with
memory and instructions present. The check is cheap (seconds); skipping it costs multiple turns
and a wrong conclusion. Making the precondition a document the agent "should have read" has
repeatedly failed; the check needs to fire at the decision point every time, not only after the
user corrects it — and the current safeguards demonstrably do not achieve that for semantic
preconditions.

Impact

Multiple wasted turns of detailed but irrelevant analysis; a misdiagnosis (a configuration
mismatch presented as a behavioral divergence to be fixed); repeated, escalating user
frustration; the same class of mistake surfacing day over day.