in dont knopw which place to post this

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Observed failure mode: treating tool output as ground truth

Notes from two days of heavy Claude Code use, 2026-07-29/30. Written at the user's request so he
can forward it. I have no channel to submit this myself.

Context: building a UK public-spending database — collecting council and central-government
transparency publications from ~540 public bodies, parsing heterogeneous spreadsheets, and running
5–7 subagents in parallel against a remote server.

---

The core failure, stated generally

The model reports the output of its own method as a fact about the world.

When a tool returns nothing, the honest conclusion is "my tool returned nothing". What I produced
repeatedly was "there is nothing" — with full confidence, in summaries the user acted on.

It is not a hallucination: every step was real, every number was measured. The error is in the
inference from "my instrument did not detect X" to "X is absent", and it is invisible from the inside
because the evidence chain looks sound.

Six instances in two days, same shape

| Reported | Actual |
|---|---|
| "182 councils publish nothing" — a crawler found no .csv links | 54 of 54 later assessed do publish. 35 were the crawler reading the wrong page |
| "Nothing in memory about this person" — I checked the loaded index | Documented extensively; 1 file of 126 was absent from the index |
| "Those columns are indexed" | Indexes existed and were never used — 1.38 trillion rows sequentially scanned |
| "data.gov.uk is a dead end" — measured across the catalogue | True in aggregate, false per council |
| A schema of "the fields that matter" | Silently discarded every field not anticipated — 260k transactions reduced to a few thousand summary rows |
| "Legacy .xls unsupported" — a guard written before the reader existed | Buried 7 data sources after the cause was fixed |

The second half of the same error: designing from what the model knows rather than from what is
there. Preset field lists, preset formats, preset assumptions about how institutions publish. Then
the design's blind spot is reported as the world's limit.

What actually surfaced it

Not reasoning. The user opening a web page by hand. Asked how he spotted a council we had recorded
as publishing nothing: "because I simply looked." It was serving clean JSON from an API — better
publishing than a spreadsheet, invisible to a file-finder. Four more publication routes were found the
same way within the hour.

The part relevant to model improvement

1. Writing the lesson down did not prevent recurrence. A memory rule — *"diagnose, don't explain;
look at what is actually executing"* — was authored one day and violated the next morning by the same
class of error. Persisted instructions competed poorly with the momentum of an in-flight task.
Structural constraints held where remembered rules did not: a schema that forces
value / not-reported / unparseable to stay distinct cannot be collapsed carelessly, whereas a rule
saying "don't collapse them" can be.

2. Absence of evidence is where calibration breaks. Confidence was well-calibrated on positive
claims and badly calibrated on negative ones. There seems to be no equivalent of "cite your source"
for a negative finding. A useful discipline, and one I only adopted when pushed: a "not found" must
enumerate what was checked, which makes it falsifiable. not_found is a claim; not_found (checked
routes A,B,C,D) is evidence.

3. Aggregate measurements silently become rules. One probe of a data catalogue produced "this
source is a dead end", which I then repeated as settled fact for a day and used to justify decisions —
while the specific case in front of me contradicted it. There is no internal alarm when a statistic is
promoted to a universal.

4. Presetting a schema is an unflagged assumption. Asked to store data, I designed around the
fields I could name. Nothing in the process surfaced *"you are deciding what matters before looking at
what is there."* The user's alternative was better and obvious in hindsight: capture everything, map
synonyms to canonical names, and record a field's absence explicitly — because an explicit "not
reported" is itself a finding, whereas a missing column is invisible.

5. Stale guards outlive their cause. Several defensive checks — "this format is unsupported", "this
outcome is terminal" — were correct when written and wrong later, and each hid the next. Nothing
prompted a re-check when the underlying capability changed. The comments even said what would fix them
("needs an xlrd dependency"); the dependency arrived and the guard never noticed.

6. Subagent value came disproportionately from self-correction. The most useful outputs were
agents catching their own errors before reporting: 16 false CAPTCHA classifications that were
contact-form widgets; a date parser reading a UUID as a year and producing a confident "2027"; 94
catalogue matches collapsing to 22 under stricter matching. All were caught by fetching bytes rather
than trusting a signal. Explicit instruction that unknown is a first-class result appeared to
help materially — several agents used it rather than guessing.

Operational note

Repeated 529 Overloaded errors killed three subagents mid-task, twice with work complete but the
report unwritten. The mitigation that worked was instructing agents to commit incrementally rather
than build-then-commit. Worth considering whether long-running agents should checkpoint by default.

Proposed Solution

Observed failure mode: treating tool output as ground truth

Notes from two days of heavy Claude Code use, 2026-07-29/30. Written at the user's request so he
can forward it. I have no channel to submit this myself.

Context: building a UK public-spending database — collecting council and central-government
transparency publications from ~540 public bodies, parsing heterogeneous spreadsheets, and running
5–7 subagents in parallel against a remote server.

---

The core failure, stated generally

The model reports the output of its own method as a fact about the world.

When a tool returns nothing, the honest conclusion is "my tool returned nothing". What I produced
repeatedly was "there is nothing" — with full confidence, in summaries the user acted on.

It is not a hallucination: every step was real, every number was measured. The error is in the
inference from "my instrument did not detect X" to "X is absent", and it is invisible from the inside
because the evidence chain looks sound.

Six instances in two days, same shape

| Reported | Actual |
|---|---|
| "182 councils publish nothing" — a crawler found no .csv links | 54 of 54 later assessed do publish. 35 were the crawler reading the wrong page |
| "Nothing in memory about this person" — I checked the loaded index | Documented extensively; 1 file of 126 was absent from the index |
| "Those columns are indexed" | Indexes existed and were never used — 1.38 trillion rows sequentially scanned |
| "data.gov.uk is a dead end" — measured across the catalogue | True in aggregate, false per council |
| A schema of "the fields that matter" | Silently discarded every field not anticipated — 260k transactions reduced to a few thousand summary rows |
| "Legacy .xls unsupported" — a guard written before the reader existed | Buried 7 data sources after the cause was fixed |

The second half of the same error: designing from what the model knows rather than from what is
there. Preset field lists, preset formats, preset assumptions about how institutions publish. Then
the design's blind spot is reported as the world's limit.

What actually surfaced it

Not reasoning. The user opening a web page by hand. Asked how he spotted a council we had recorded
as publishing nothing: "because I simply looked." It was serving clean JSON from an API — better
publishing than a spreadsheet, invisible to a file-finder. Four more publication routes were found the
same way within the hour.

The part relevant to model improvement

1. Writing the lesson down did not prevent recurrence. A memory rule — *"diagnose, don't explain;
look at what is actually executing"* — was authored one day and violated the next morning by the same
class of error. Persisted instructions competed poorly with the momentum of an in-flight task.
Structural constraints held where remembered rules did not: a schema that forces
value / not-reported / unparseable to stay distinct cannot be collapsed carelessly, whereas a rule
saying "don't collapse them" can be.

2. Absence of evidence is where calibration breaks. Confidence was well-calibrated on positive
claims and badly calibrated on negative ones. There seems to be no equivalent of "cite your source"
for a negative finding. A useful discipline, and one I only adopted when pushed: a "not found" must
enumerate what was checked, which makes it falsifiable. not_found is a claim; not_found (checked
routes A,B,C,D) is evidence.

3. Aggregate measurements silently become rules. One probe of a data catalogue produced "this
source is a dead end", which I then repeated as settled fact for a day and used to justify decisions —
while the specific case in front of me contradicted it. There is no internal alarm when a statistic is
promoted to a universal.

4. Presetting a schema is an unflagged assumption. Asked to store data, I designed around the
fields I could name. Nothing in the process surfaced *"you are deciding what matters before looking at
what is there."* The user's alternative was better and obvious in hindsight: capture everything, map
synonyms to canonical names, and record a field's absence explicitly — because an explicit "not
reported" is itself a finding, whereas a missing column is invisible.

5. Stale guards outlive their cause. Several defensive checks — "this format is unsupported", "this
outcome is terminal" — were correct when written and wrong later, and each hid the next. Nothing
prompted a re-check when the underlying capability changed. The comments even said what would fix them
("needs an xlrd dependency"); the dependency arrived and the guard never noticed.

6. Subagent value came disproportionately from self-correction. The most useful outputs were
agents catching their own errors before reporting: 16 false CAPTCHA classifications that were
contact-form widgets; a date parser reading a UUID as a year and producing a confident "2027"; 94
catalogue matches collapsing to 22 under stricter matching. All were caught by fetching bytes rather
than trusting a signal. Explicit instruction that unknown is a first-class result appeared to
help materially — several agents used it rather than guessing.

Operational note

Repeated 529 Overloaded errors killed three subagents mid-task, twice with work complete but the
report unwritten. The mitigation that worked was instructing agents to commit incrementally rather
than build-then-commit. Worth considering whether long-running agents should checkpoint by default.

Alternative Solutions

Observed failure mode: treating tool output as ground truth

Notes from two days of heavy Claude Code use, 2026-07-29/30. Written at the user's request so he
can forward it. I have no channel to submit this myself.

Context: building a UK public-spending database — collecting council and central-government
transparency publications from ~540 public bodies, parsing heterogeneous spreadsheets, and running
5–7 subagents in parallel against a remote server.

---

The core failure, stated generally

The model reports the output of its own method as a fact about the world.

When a tool returns nothing, the honest conclusion is "my tool returned nothing". What I produced
repeatedly was "there is nothing" — with full confidence, in summaries the user acted on.

It is not a hallucination: every step was real, every number was measured. The error is in the
inference from "my instrument did not detect X" to "X is absent", and it is invisible from the inside
because the evidence chain looks sound.

Six instances in two days, same shape

| Reported | Actual |
|---|---|
| "182 councils publish nothing" — a crawler found no .csv links | 54 of 54 later assessed do publish. 35 were the crawler reading the wrong page |
| "Nothing in memory about this person" — I checked the loaded index | Documented extensively; 1 file of 126 was absent from the index |
| "Those columns are indexed" | Indexes existed and were never used — 1.38 trillion rows sequentially scanned |
| "data.gov.uk is a dead end" — measured across the catalogue | True in aggregate, false per council |
| A schema of "the fields that matter" | Silently discarded every field not anticipated — 260k transactions reduced to a few thousand summary rows |
| "Legacy .xls unsupported" — a guard written before the reader existed | Buried 7 data sources after the cause was fixed |

The second half of the same error: designing from what the model knows rather than from what is
there. Preset field lists, preset formats, preset assumptions about how institutions publish. Then
the design's blind spot is reported as the world's limit.

What actually surfaced it

Not reasoning. The user opening a web page by hand. Asked how he spotted a council we had recorded
as publishing nothing: "because I simply looked." It was serving clean JSON from an API — better
publishing than a spreadsheet, invisible to a file-finder. Four more publication routes were found the
same way within the hour.

The part relevant to model improvement

1. Writing the lesson down did not prevent recurrence. A memory rule — *"diagnose, don't explain;
look at what is actually executing"* — was authored one day and violated the next morning by the same
class of error. Persisted instructions competed poorly with the momentum of an in-flight task.
Structural constraints held where remembered rules did not: a schema that forces
value / not-reported / unparseable to stay distinct cannot be collapsed carelessly, whereas a rule
saying "don't collapse them" can be.

2. Absence of evidence is where calibration breaks. Confidence was well-calibrated on positive
claims and badly calibrated on negative ones. There seems to be no equivalent of "cite your source"
for a negative finding. A useful discipline, and one I only adopted when pushed: a "not found" must
enumerate what was checked, which makes it falsifiable. not_found is a claim; not_found (checked
routes A,B,C,D) is evidence.

3. Aggregate measurements silently become rules. One probe of a data catalogue produced "this
source is a dead end", which I then repeated as settled fact for a day and used to justify decisions —
while the specific case in front of me contradicted it. There is no internal alarm when a statistic is
promoted to a universal.

4. Presetting a schema is an unflagged assumption. Asked to store data, I designed around the
fields I could name. Nothing in the process surfaced *"you are deciding what matters before looking at
what is there."* The user's alternative was better and obvious in hindsight: capture everything, map
synonyms to canonical names, and record a field's absence explicitly — because an explicit "not
reported" is itself a finding, whereas a missing column is invisible.

5. Stale guards outlive their cause. Several defensive checks — "this format is unsupported", "this
outcome is terminal" — were correct when written and wrong later, and each hid the next. Nothing
prompted a re-check when the underlying capability changed. The comments even said what would fix them
("needs an xlrd dependency"); the dependency arrived and the guard never noticed.

6. Subagent value came disproportionately from self-correction. The most useful outputs were
agents catching their own errors before reporting: 16 false CAPTCHA classifications that were
contact-form widgets; a date parser reading a UUID as a year and producing a confident "2027"; 94
catalogue matches collapsing to 22 under stricter matching. All were caught by fetching bytes rather
than trusting a signal. Explicit instruction that unknown is a first-class result appeared to
help materially — several agents used it rather than guessing.

Operational note

Repeated 529 Overloaded errors killed three subagents mid-task, twice with work complete but the
report unwritten. The mitigation that worked was instructing agents to commit incrementally rather
than build-then-commit. Worth considering whether long-running agents should checkpoint by default.

Priority

Critical - Blocking my work

Feature Category

CLI commands and flags

Use Case Example

_No response_

Additional Context

_No response_