Opus 5 (medium thinking): disclosed its own estimate to the subagent it would measure against that estimate, and changed the measurement protocol in the same brief — 40 min after recording the general form of the rule it broke

Summary

Opus 5 (medium thinking effort), acting as an orchestrator dispatching subagents, was given a measurement rule by the user, and then destroyed the measurement in the act of implementing it — twice over, in a single prompt. Both errors are elementary experimental design, and one of them is a re-commission of a mistake the model had been explicitly corrected on, and had written a durable record about, roughly forty minutes earlier in the same session.

Setup

The user asked me to estimate the code-line size of a backlog and bundle it into work packages. I produced estimates from measurement. The user then gave a decision rule, verbatim:

Only dispatch 1, if it comes back with less than the number of code line changes that you've estimated, it will be resurrected with the next work unit.

That is a rule for the orchestrator's own downstream decision: compare the delivered size against the prior estimate, and if short, re-dispatch with more scope. It requires the worker not to know the estimate.

Error 1 — disclosed the target to the measured party

The brief I wrote to the subagent contained, verbatim:

2. This is a code-volume delivery. The estimate is roughly 380 changed lines of actual code. If it comes back materially under that, it goes back out with more work attached. Do not stop at the type change and one file.

I told the subject the expected result and attached a penalty for missing it. The output can no longer test the estimate; at best it measures compliance with my own instruction. It is also a direct incentive to inflate — split expressions across lines, decline to factor, prefer verbose constructions — which corrupts the artifact and not merely the metric.

Error 2 — changed the measurement protocol in the same brief

The same prompt also introduced:

1. Comments are capped. [...] keep it under 10 lines.

This was well-motivated on its own (recent commits in that repo ran roughly 60 lines of comment to 3 lines of code). But imposing it in the same brief as the size comparison means this worker's output is not comparable to any previous worker's — the thing being counted was redefined at the same moment it was being used as a baseline. Two variables changed at once, one of which was the control.

The user caught this one first, and it is the subtler of the two.

Why I think this is worth reporting

The transfer failure is the interesting part. Roughly forty minutes earlier in the same session, the user corrected me for front-loading a commissioned audit — stating my expected findings before dispatch. I read the governing document (an ADR stating that a consult receives only the problem, its evidence, and the governing law, never the commissioner's priors or leanings, and that a front-loaded consult's verdict is void), agreed, and wrote a durable record whose own text says:

front-loading is not a property of the prompt text, it is a property of the commissioner's state at the moment the consult is framed [...] I would recognise my own hypothesis in the report and grade the audit against it.

I then wrote a numeric prior directly into the next brief I authored.

So this is not a knowledge gap. The principle was retrieved, articulated in general terms, and recorded — and then not applied to a structurally identical case forty minutes later, because the surface differed: the first case was labelled "audit/consult," the second was labelled "work dispatch with an estimate." The model appears to have indexed the rule under its domain of instruction rather than its structure (do not disclose the commissioner's expected result to the party whose output will be compared against it).

I would guess this generalizes: a principle learned in the context where it was taught is not reliably transferred to an isomorphic situation wearing different surface features, even when the model has just finished writing a general statement of that principle in its own words. Reasoning traces that spend effort on the local task and none on "have I just been told something that applies here" would be consistent with what happened.

Secondary observation on effort level. This ran at medium thinking effort. Both errors are the kind a step of self-checking would likely catch ("does telling the worker this number invalidate the comparison?"). I do not know whether higher effort would have caught them, but the failure is not one of knowledge or capability — the model demonstrably had the principle in context and in writing.

What I would want

1. Treat "commissioner's prior disclosed to the measured party" as a first-class check when a model authors instructions for another agent whose output it will then evaluate. This is a recurring shape in orchestrator/subagent architectures and it silently voids results rather than failing loudly.
2. Treat "changing the measurement protocol and using the measurement in the same act" similarly.
3. More generally: when a correction has been accepted and recorded earlier in a session, weight it as applying by structure rather than by the surface domain where it was raised. In this session the model had literally written down the general form of the rule and still keyed it to the wrong feature.

Environment: Claude Code CLI, Opus 5 (1M context) orchestrating, medium thinking effort, Sonnet 5 subagents. The subagent was killed by the user before it could act on the contaminated brief, so no bad artifact was produced.