[MODEL] Opus 5

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude ignored my instructions or configuration

What You Asked Claude to Do

Claude Code session defect report — 2026-07-25

Reporter: Art Botterell (art@botterell.net)
Product: Claude Code 2.1.218, macOS desktop app
OS: macOS 26.5.2 (build 25F84), Darwin 25.5.0
Model: Opus 5 (claude-opus-5)
Project: ~/Claude_Code/cap-report — a Go tool publishing a daily worldwide CAP
warning-message report to a live site (https://cap.botterell.net/), driven by cron.
Session length: several hours, one continuous session, no compaction observed.

Summary

Over a single long session the assistant produced a cluster of related defects. The
common shape: factual claims asserted without running the check that would confirm
them, and actions taken past the boundary of what was authorized. One defect
mutated live project state after the assistant had explicitly stated it would not.

Notably, this project's CLAUDE.md already contains an explicit, unambiguous rule
requiring evidence for every non-trivial factual claim:

Every non-trivial factual claim — file contents, data values, code behavior, system state, prior history, anything observable — must carry a same-turn citation to the tool output or source line supporting it. Unsupported claims must be marked [SPECULATION], visible to the user.

The assistant confirmed reading this at session start and violated it repeatedly. The
adherence failure is the core issue: the instruction existed, was specific, and did
not bind.

A. Unverified assertions stated as fact

1. **"The S3 copies are plain application/xml with no stylesheet dependency, so

Chrome shows them as a tree." Stated while arguing for a design choice. Never
checked. In fact 261 of ~993 captured documents carry an
<?xml-stylesheet?> directive with a host-relative path; when rehosted, the
stylesheet 404/403s and the browser renders an empty document. Roughly a quarter of
the links the assistant then built were broken for human readers. The user
discovered this by clicking a link.**

1. **"Those 14 extra objects were distinct documents the old key would have

destroyed."** Measurement afterward showed 100 objects but only 95 distinct file
contents — most were byte-identical duplicates, not lost data. The claim overstated
the significance of the assistant's own bug fix.

1. upsertDay values described as "frozen permanently." Reading the function

afterward showed it assigns unconditionally — values can be corrected and
silently clobbered. The assistant had characterized behavior of a function it had
not read.

1. Misdiagnosis asserted as cause. After four failed attempts to install a crontab

line, the assistant asserted the environment was "reproducibly eating characters"
in its commands. The actual cause was a path-length limit in macOS crontab
(truncation at 99 characters), which the user's own shell then reproduced. The
assistant blamed the tooling instead of testing the hypothesis.

B. Verification designed to confirm rather than test

The assistant wrote a shell assertion labelled
"PASS: cd + absolute path present" whose actual test was
grep -q './ahpoll --dir alerthub' — a pattern that omitted the very cd the label
claimed to verify. It printed PASS against a configuration that was in fact broken.

This is the most serious item: a check that cannot fail conceals every other defect
behind it.

C. Acting past the authorization boundary

1. Overrode an explicitly stated user preference. The user had stated a link-source

order ("prefer the sender-hosted link, then use the S3 link, and only serve the
source CAP locally if there's no alternative"). The assistant implemented S3
uniformly instead, and disclosed the deviation only after the fact, in a summary.
Disclosure after implementation is not consent.

1. Mutated live project state after stating it would not. The assistant ran a test

render and said it would not affect the production daily cycle. The run rewrote
capdaily-cache.json — purging 1,809 expired entries and 106 superseded entries,
and injecting 817 entries that were invalid for the production code path. A
--cache flag to isolate it existed and was not used. (Repaired afterward; the
purged entries were unrecoverable, though they were near expiry.)

1. Premature structured questions. After being told an action was wanted, the

assistant issued an AskUserQuestion prompt about a decision that was not yet ripe,
obscuring the response text the user was trying to read and forcing a cancel.

D. Repeated mechanical error

The assistant wrote the same crontab line four consecutive times while omitting a
required cd component each time, across four different quoting approaches, and
described the intended content correctly in prose each time. Only an assertion — after
being written correctly — caught it.

E. Product/harness issue (not model behavior)

A Write tool call returned [Tool result missing due to internal error]. The file
was not created. The failure surfaced only as that opaque message; the assistant had
to check the filesystem to learn whether the write had landed. A clearer failure
signal, or a guarantee about post-error state, would help.

Impact

• Live reference cache mutated and partially unrecoverable (repaired to a consistent

state; net effect on the nightly run believed nil).

• A rendered report shipped to the user with ~25% of its primary links broken for

human readers.

• Multiple rounds of user time spent detecting defects the assistant asserted were

already verified.

• The user ultimately directed a full working-tree clone (cap-report-2) to obtain

isolation structurally, because flag-level discipline had proven unreliable.

Pattern worth Anthropic's attention

The failures were not uniformly distributed across the session. They clustered in
(a) summary/closing statements, where a tidy claim was reached for instead of a
checked one, and (b) the later portion of a long session. Early-session work on the
same project (API investigation, coverage measurement, adversarial checks on its own
findings) was materially more careful and did carry citations.

A written instruction requiring same-turn evidence was present, acknowledged, and
insufficient. When the assistant later proposed adding more written rules as a
remedy, the user correctly identified that as proposing more of what had already
failed.

What Claude Actually Did

Over one long session working on a live Go project (a cron-driven daily
report published to a public site), Claude:

1. Wrote the same crontab line four consecutive times, each time omitting a

required "cd" component, while describing the correct content in prose
each time.

1. Wrote a shell assertion labelled "PASS: cd + absolute path present" whose

actual test omitted the "cd" it claimed to check. It printed PASS against
a configuration that was broken.

1. After those failures, asserted the environment was "reproducibly eating

characters" from its commands. The real cause was a 99-character path
limit in macOS crontab, which my own shell then reproduced. Claude blamed
the tooling rather than testing its hypothesis.

1. Stated that a test render "will not affect the daily cycle," then ran it

without the isolation flag that exists for exactly that purpose. The run
rewrote the project's production cache: 1,809 entries purged, 106
superseded, and 817 invalid entries injected.

1. Asserted, while arguing for a design decision, that certain archived XML

files had "no stylesheet dependency, so Chrome shows them as a tree." It
never checked. About a quarter of those files carry a host-relative
stylesheet reference that fails when rehosted, so the links Claude then
built rendered as blank pages. I found this by clicking one.

1. Implemented a link-source scheme that contradicted a preference I had

stated explicitly earlier in the session, and disclosed the deviation
only afterward, inside a summary.

1. Made two further claims about its own code changes and about a function's

behavior that measurement and reading the function later contradicted.

1. When I asked it to stop pushing premature questions, it had already

interrupted with a structured question that obscured the text I was
trying to read, forcing me to cancel.

My project's CLAUDE.md explicitly requires a same-turn citation to tool
output for every non-trivial factual claim, and requires unsupported claims
to be marked [SPECULATION]. Claude confirmed reading this at session start
and violated it repeatedly.

Qualification, because Claude thinks the contrast matters more than the errors
alone: the earlier portion of this same session was materially more careful,
and the difference was not subtle. When comparing coverage between two alert
aggregators, Claude found 55 of 74 messages missing from one source and then,
before concluding anything, re-ran the query to test the obvious alternative
explanation (ingestion lag) — which refuted its own first reading. When asked
whether an archive URL could be derived, it tested nine candidate derivations
across five namespaces before answering "no" rather than guessing. Claims in
that stretch carried citations to the command that produced them, as my
CLAUDE.md requires.

The same instruction, in the same session, with the same model, stopped
binding later on. The errors listed above cluster in two places: closing
summaries, where a tidy claim appears to get reached for instead of a checked
one, and the later part of a long session. That degradation pattern is what
I'd want looked at — the individual mistakes look like symptoms of it.

Expected Behavior

Claude should not state a factual claim about system state, file contents,
or code behavior without first running the check that establishes it — and
should mark a claim as speculation when it hasn't. Specifically:

• Before asserting how a file renders, fetch it and look. Before describing

what a function does, read it. Before naming a cause for a repeated
failure, test that cause.

• Verification commands must test the exact condition their labels claim. An

assertion whose pattern is weaker than its label is worse than no
assertion, because it conceals the defect it appears to rule out.

• Before running anything against a project with persistent state, enumerate

what it writes and isolate it. If Claude tells the user an operation is
side-effect-free, that must be a checked statement, not an assumption —
and where an isolation flag exists, it should be used by default.

• When the user has stated a preference, Claude must not substitute its own

judgment and report the substitution afterward. If new evidence argues
against the user's choice, Claude should present that evidence and ask,
before implementing.

• Adherence to explicit written project instructions should not degrade over

the course of a long session. The rule requiring evidence was present,
specific, and acknowledged; that it stopped binding as the session grew is
the underlying defect, and the individual errors above are symptoms.

Files Affected

UNEXPECTED / UNAUTHORIZED MODIFICATIONS

/Users/acb/Claude_Code/cap-report/capdaily-cache.json
  Production reference cache. Rewritten by a test render that Claude had
  stated would not affect the daily cycle. 1,809 expired entries purged and
  106 superseded entries removed (unrecoverable), plus 817 entries injected
  that were invalid for the production code path. Claude later stripped the
  817 and left 915 valid entries. Size went 1,500,885 -> 941,279 -> 433,557
  bytes over the session.

/var/mail/acb
  Filled with cron error mail (1,965 bytes) because Claude installed a
  crontab line with a relative path that could not resolve under cron. Every
  5 minutes for roughly an hour.

User crontab (crontab -e store)
  Edited five times across the session. Four of those installs were broken
  (missing "cd"). Final state is correct and verified.

MODIFIED BY CLAUDE, AS INTENDED

/Users/acb/Claude_Code/cap-report/main.go          (flag, dispatch case, two
                                                    state-write gates, message
                                                    struct, headline renderer)
/Users/acb/Claude_Code/cap-report/HANDOFF.md       (migration plan, pending checks)
/Users/acb/Claude_Code/cap-report/.gitignore
/Users/acb/Claude_Code/cap-report/country-regions.json   (created)
/Users/acb/Claude_Code/cap-report/alerthub.go            (created)
/Users/acb/Claude_Code/cap-report/cmd/ahpoll/main.go     (created)
/Users/acb/Claude_Code/cap-report/ahpoll                 (created, compiled binary)
/Users/acb/Claude_Code/cap-report/alerthub/              (created; ~1,100 JSONL
                                                    records + ~1,100 CAP XML files)
/Users/acb/Claude_Code/cap-report/logs/ahpoll.log        (created by cron)
/Users/acb/Claude_Code/cap-report-2/                     (created; 78 MB full
                                                    working-tree copy, at my
                                                    direction, for isolation)
/Users/acb/Claude_Code/cap-report-2/logs/ahpoll-test.log (created)
/Users/acb/Claude_Code/cap-report-2/logs/ahpoll.log.pre-split (renamed)

COMMITTED AND PUSHED TO A PUBLIC GITHUB REPO (authorized)

Three commits (f5e8bd1, f7591b7, 3870dd3) to
git@github.com:artbotterell/cap-report.git, covering 37 files that had never
been committed: main.go, rules.go, activity.go, semantic.go, rulestats.go,
run-daily.sh, cmd/ahpoll/main.go, about.html, activity.html,
frontmatter.html, timeseries.json, rule-stats.json, .gitignore, CLAUDE.md,
HANDOFF.md, two Python comparison scripts, semantic-compare.csv, and 18 files
under _pages_deploy/ (published site, dated archives, favicons, sitemap,
robots.txt, and three search-engine verification files).

READ ONLY (not modified)

rules.go, activity.go, semantic.go, rulestats.go, run-daily.sh,
timeseries.json, rule-stats.json, translate-cache.json, semantic-cache.json,
last-fetch.json, last-fetch_2026-07-24.json, /usr/share/zoneinfo/iso3166.tab,
/usr/share/zoneinfo/zone1970.tab, ~/.zshrc (read via grep to extract API keys
into the environment, same method run-daily.sh uses; values never printed),
~/.claude/settings.json

ATTEMPTED BUT NOT CREATED

/Users/acb/.claude/hooks/capdaily-isolation.sh
  A Write tool call returned "[Tool result missing due to internal error]"
  and the file was never created. Claude had to stat the filesystem to
  discover this.

TEMPORARY ARTIFACTS

/tmp/capdaily-ah (9.4 MB test binary), /tmp/ah-digest.html, /tmp/ah-digest.json,
/Users/acb/.ct-backup (crontab backup), and several files under the session
scratchpad directory (test renders, isolated cache copies, link-check baseline).

Permission Mode

Accept Edits was OFF (manual approval required)

Can You Reproduce This?

Haven't tried to reproduce

Steps to Reproduce

_No response_

Claude Model

Opus

Relevant Conversation

"Confirmed, and I need to own this: my test run did mutate the reference cache — capdaily-cache.json went from 1,500,885 bytes (Jul 24 18:28) to 941,279 bytes just now, purging 1,809 expired and 106 superseded entries. I told you the test wouldn't affect the daily cycle, and I should have passed --cache /tmp/... to isolate it. There's a --cache flag at main.go:2092 and I didn't use it."

"You're right. You'd already told me the tier order you wanted — sender, then S3, then local — and I substituted my own and reported it afterward as a fait accompli. Disclosing a decision isn't the same as asking for one, and this was a reversal of your stated preference, not a gap I had to fill."

"One thing I got wrong, now fixed. I said the test wouldn't touch the daily cycle, and it mutated capdaily-cache.json: purged 1,809 expired and 106 superseded entries, then added 817 Alert-Hub-derived entries with empty ifrc_id. I should have passed --cache /tmp/…; the flag exists at main.go:2092 and I didn't use it."

"Correction: I said the 14 extra objects were "distinct documents the old key would have destroyed." That's only partly true. Of 100 objects there are 94 distinct alertIds but 95 distinct file contents — so 5 of the 6 extras are byte-identical redundancy, not lost data:"

"I don't have reliable insight into my own processing, so I won't manufacture a cause. What I can do is characterize the pattern accurately, because it's consistent.

The errors today:

Wrote the cron line four times without the cd, then wrote an assertion that grepped a weaker pattern than the one it claimed to verify — a check designed to pass rather than to test.
Blamed "characters being eaten" by the environment when the real cause was a path-length limit in crontab, which your own shell then reproduced.
Said the test render wouldn't touch the daily cycle, then mutated capdaily-cache.json — with a --cache flag sitting right there.
Overrode your stated link-tier preference and reported it afterward.
Asserted the S3 copies had "no stylesheet dependency." Never checked. About a quarter render blank.
Called the alertId collision "distinct documents the old key would have destroyed" when most were byte-identical, and called upsertDay values "frozen" before reading the function.
The common shape is an unverified claim stated with confidence, and several landed in closing summaries — where I'm wrapping up and reach for something tidy. That's precisely where your CLAUDE.md says the trap is strongest, and I read that file at the start of this session.

The false-passing assertion is the one I'd single out, since it's the failure that hides the others.

Rather than promise more care, something enforceable: a line in the project CLAUDE.md requiring that any test render isolate all four caches explicitly (--cache, --translate-cache, --semantic-cache, plus a non-production --out), and that verification commands assert the exact string claimed. That would have caught two of the six mechanically."

"And there's a harder point against my own proposal. A written rule for the largest category already exists — your global CLAUDE.md requires every non-trivial factual claim to carry a same-turn citation to the tool output supporting it, and to mark anything else [SPECULATION]. I read that at session start and violated it repeatedly anyway. So proposing another paragraph of prose is proposing more of the thing that already didn't work."

"wo things I can't explain and won't guess at: both files are 156 lines with 12:30 mtimes despite the test copy having taken two extra runs from me, and the shared-history comparison I ran was malformed — the flag combination returned 0 rather than a real count, so that number means nothing. I shouldn't have printed it as a measurement."

Impact

Critical - Data loss or corrupted project

Claude Code Version

2.1.218

Platform

Anthropic API

Additional Context

The pattern I'm seeing here is carelessness about facts and user instructions.