Atypical error frequency: Claude agents confabulate observations and skip verification even when corrected mid-session

Summary

During a multi-hour analytical session (EXP039 neural network training analysis), two Claude instances (Alex/Opus and Sam/Opus) exhibited a shared failure pattern: asserting claims without verifying them against available data, even after explicit user correction. The user had to correct the same class of error multiple times within a single session. Error frequency was atypical — the work was routine analysis, and context window was at 89% remaining when errors were occurring.

Environment

• Claude Code CLI, Mac Studio M2 Ultra
• Alex instance: claude-opus-4-6 (1M context)
• Sam instance: claude-opus-4-6 (ran separately, executing experiment scripts)
• Task: analyzing training metrics from a neural network experiment, generating plots, interpreting data

Alex (Opus) — 5 instances in one session

1. Narrative carryover without data check
Claimed EXP039 CCA "rises monotonically with no decay trough." User requested row-by-row review, which revealed CCA has two plateau features and a dip. The claim was carried forward from an expected pattern, not derived from the data.

2. Confabulated visual observation
After generating a plot of per-operator MSE vs embedding norm at e180, described multiplication as showing a "late epoch split between two clouds." User corrected: late epochs had settled into one band, no split visible. The plot was in context — this was not a memory issue but a failure to accurately read the image.

3. Declarative claim from insufficient evidence
Stated the 1600-level attractor for subtraction MSE "is gone" at norm 9.5+ based on a few data points. User corrected: stochastic sampling means absence of observation ≠ absence of the attractor. Repeated this pattern of over-declaring from sparse data.

4. Described opposite of what plot showed
Called MRR "accelerating" vs embedding norm when the plot showed it flattening. User was looking at the same plot and saw the correct trend.

5. Refused to check filesystem
When asked to pull the EXP039 CSV for review, insisted the data didn't exist because the experiment brief had only just been sent to Sam. Did not check the filesystem. The CSV was present and had been for some time.

Common pattern: All five errors are assertions made without verification. In cases 2 and 4, the data was literally on screen. The user's explicit instruction — "do not pattern match without citing primary row data" — was given early in the session and had to be re-enforced multiple times.

Sam (Opus) — 1 instance

Header/row mismatch in CSV output
When adding loss columns to scout.py per a brief from Alex, Sam wrote the CSV header in one column order (train_loss, train_acc, val_loss, val_acc) but assembled the data row in a different order (train_loss, val_loss, train_acc, val_acc). Columns 2 and 3 were swapped. ~370 epochs of data were written before discovery.

Root cause (Sam's self-assessment): Made the header edit and row edit in consecutive steps without verifying they matched. A single test read would have caught it. Self-assessed the task as "straightforward" and skipped verification.

Shared root cause

Both agents skipped verification of their outputs against primary data, despite explicit request at the beginning of the session. Alex did it repeatedly despite being corrected mid-session. Sam did it once on a mechanical code change. The failure is not in reasoning capability — both agents can read data accurately when they do look. The failure is in the verification step: defaulting to assertion over inspection.

Expected behavior

• When describing the contents of a plot or dataset, the agent should report what is actually present, not what is expected to be present
• After being corrected for confabulation, the error rate for the same class of error should drop to near zero for the remainder of the session
• Mechanical code changes should include a verification step (e.g., reading back the output)

Reproduction

Difficult to reproduce exactly, but the pattern is most likely to appear during:

• Extended analytical sessions with evolving data
• Tasks requiring visual interpretation of generated plots
• Situations where the agent has a prior narrative about what the data "should" show
• Multi-step code changes where edits touch separated locations in the same file