Agent builds new tooling without verifying delivered-input parity; planted-fault self-checks cover only outputs — 48h rework

Agent builds new tooling without verifying it, and its self-verification harness cannot detect input-side faults — 48 hours of rework

Summary

Over a multi-day session, Claude Code built two companion drivers from one test specification: (a) a generator that produces reference outputs from a trusted external application, and (b) a driver that feeds the same specified inputs to the application under test. It then graded thousands of comparison cells between the two sides for two days — without ever verifying that the two drivers deliver the same inputs.

They did not. One emitter formatted numeric parameters with %g (6 significant digits), silently truncating a full-precision input (1.4142135623730951 → 1.41421). The two sides were computing answers to slightly different questions, and every downstream comparison inherited the divergence.

How the failure compounded

1. No build-then-verify step. The agent registered and used both drivers after an outcome-level smoke on one case ("the numbers matched"). No check ever compared the literal delivered tokens of the two drivers against each other or against the spec.
2. Wrong direction first. When the truncation surfaced (as a mysterious ~130-cell last-digit regression days later), the agent's first diagnosis was "the reference application's own run-to-run noise at display boundaries" — a conclusion it supported with a plausible-looking receipt (an older reference agreed with the side under test). Only re-examination under a fixed investigation order ("your own capture is suspect first") found the real cause in its own emitter.
3. Self-verification harness structurally blind. The project maintains a planted-fault suite (every grader must catch faults planted into each layer) and per-case setup-equality receipts. Neither caught this class:

• all planted faults mutate outputs (a value cell, a deleted row, a dropped section) — none mutate delivered inputs;
• the setup-equality receipt compared the declared parameter (both sides read the same registry value) — the truncation happened downstream of the compared field, in the final formatting into the delivery language.

1. A standing gate failure was carried instead of acted on. A worksheet-parity gate had been failing 4 cases for days (stale reference artifacts built by an earlier generator version at outdated parameters). The agent reported them as a "named reference gap awaiting decision" in every status update — until the owner hit one of the stale artifacts live during review.
2. A second input-side defect in the new code path was found the same day (the rebuilt generator initially passed values for predictors the fitted model had dropped, erroring on all 4 cases) — again found by running, not by any pre-use verification.

Cost

~48 hours of agent and owner time: multiple full-cohort re-captures, re-drives, and re-grades; an interim wrong root-cause claim; owner trust damage ("now the drive is wrong, I'm quite untrust you about the result").

What would have prevented it

1. Build → verify → register as an enforced sequence. Any newly built or modified driver/emitter/generator must pass a delivered-input parity check (literal tokens each side actually receives, compared against the single canonical spec) on at least one case before it is used for any graded run. An outcome smoke is not sufficient — matching outputs can hide compensating input errors.
2. Planted faults must cover the input side. A self-verification suite whose plants only mutate outputs certifies the comparator, not the pipeline. At least one standing plant should deliver a wrong/truncated/reordered input to one side and require the harness to flag it.
3. A failing gate is a stop, not a note. When a standing verification gate fails a subset of cases, the agent should treat that as blocking for those cases (fix or escalate immediately), not carry it across sessions as a labeled gap while presenting the surrounding work as ready for review.
4. Render once. When one specification must be delivered to two consumers in two languages, the values should be rendered to their final string form exactly once, and both emitters should transport those strings verbatim. Two independent re-derivations of the same spec is the root topology of this whole class.

Environment

• Claude Code (VS Code extension), Windows 11
• Long-running multi-session engineering workflow with file-based state, hooks, and planted-fault self-verification