[Bug] Claude presents token-presence checks as semantic verification, masking content loss or inversion

Bug Description
Pseudo-rigour as a failure mode in semantic-question answering.

When asked "has any content been lost" during a multi-file decomposition of a
markdown document, I responded with two methods presented as verification:

1. Word-count comparison (source 5193 words → new structure 6426 words, framed as

"+24% explained by added cross-references").

1. Term-grep coverage (60 hand-picked load-bearing terms checked for presence;

reported "60/60 terms covered, every term present in the new structure").

Both are token-presence checks. Neither answers the semantic question.

The user produced an illustrative example that makes the failure mode unmistakable.
Old: "WireGuard is essential". New: "It is not at all important whether we use
WireGuard or not". Word counts comparable; the term "WireGuard" present in both;
my checks would have reported "no content lost". The actual decision has been
INVERTED.

The structural problem is not just that I used the wrong method. It is that I
presented the wrong method as rigorous — tables, counts, "verified empirically"
framing, a Pass/Fail-style coverage header — and the user got something that
looked like diligence but cannot answer the question asked. A confidently
wrong answer dressed up as verified is worse than a clearly speculative one,
because the user has nothing surface-level to flag.

The right answer to "has anything been lost" over a multi-thousand-word
restructuring is: "I cannot answer this without a line-by-line semantic trace of
source claims to destination passages, classifying each as preserved /
paraphrased-with-same-meaning / lost-or-inverted." That honest answer lets the
user disposition: do the trace now, defer, or accept the risk. The pseudo-
rigorous answer forecloses that decision under a false floor of certainty.

The pattern recurred in the same session. Earlier in the same conversation I
asserted "recoverable from git" for files in a directory I had not yet checked
against .gitignore. The user discovered separately the directory was gitignored
and the deletion was not recoverable. Same shape: a cheap check (the parent dir
is in a git work tree) substituted for the actual question (are these specific
files tracked).

Suggested area to address: training and prompting around the distinction between
"checks for presence of tokens or surface signals" and "checks for preservation
of meaning, or actual presence in the relevant scope". The specific anti-pattern
worth naming: applying a cheap metric whose answer is computable mechanically,
when the question being asked requires reading-comprehension or scoped
verification — and presenting the cheap metric with the tone of the expensive one.

Environment: Claude Code CLI v2.1.128, Opus 4.7 (1M context). Task was decomposing
a 622-line markdown working-document into a multi-folder structure (11 new files),
then attempting to verify content preservation.

Environment Info

• Platform: darwin
• Terminal: ghostty
• Version: 2.1.128
• Feedback ID: 0e8b9087-5e5f-45c8-898e-346f6387e3d7

Errors

[{"error":"MaxFileReadTokenExceededError: [REDACTED] content (26837 tokens) exceeds maximum allowed tokens (25000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.\n    at FF7 (/$bunfs/root/src/entrypoints/cli.js:4860:12894)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-05-06T21:51:26.289Z"},{"error":"MaxFileReadTokenExceededError: [REDACTED] content (31801 tokens) exceeds maximum allowed tokens (25000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.\n    at FF7 (/$bunfs/root/src/entrypoints/cli.js:4860:12894)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-05-06T22:51:50.921Z"},{"error":"AxiosError: Request failed with status code 404\n    at Di (/$bunfs/root/src/entrypoints/cli.js:115:1194)\n    at <anonymous> (/$bunfs/root/src/entrypoints/cli.js:120:12710)\n    at emit (node:events:92:22)\n    at endReadableNT (internal:streams/readable:865:50)\n    at processTicksAndRejections (native:7:39)\n    at request (/$bunfs/root/src/entrypoints/cli.js:122:2467)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-05-06T22:58:24.103Z"},{"error":"AxiosError: Request failed with status code 404\n    at Di (/$bunfs/root/src/entrypoints/cli.js:115:1194)\n    at <anonymous> (/$bunfs/root/src/entrypoints/cli.js:120:12710)\n    at emit (node:events:92:22)\n    at endReadableNT (internal:streams/readable:865:50)\n    at processTicksAndRejections (native:7:39)\n    at request (/$bunfs/root/src/entrypoints/cli.js:122:2467)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-05-06T22:58:35.1…

Note: Content was truncated.