[BUG]

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

This is not a complaint about one isolated hallucination. It is a documented pattern of repeated factual and self-audit failures in Claude Cowork, using Opus 5 on High, during a deterministic data investigation.

Claude repeatedly:

• generalized from two checked documents to a third unchecked one;
• wrote a remembered timestamp (21:31) instead of rereading the source metadata, which showed 21:40;
• claimed corpus-wide absence after examining only part of the corpus;
• presented an unsupported causal inference as fact;
• changed the denominator of a percentage calculation and reported 99.2%, although the correct same-source, same-date calculation was 99.147%, which rounds to 99.1% at one decimal place.

I then created unusually strict project controls: canonical instructions, mandatory full rereading, initial and final SHA-256 hashes, explicit FACT/CALCULATION/INFERENCE/UNVERIFIED labels, enumerated-universe requirements for absence claims, fixed rounding rules, protected reference files, fresh conversations with handoffs, and a mandatory ten-item delivery gate.

Claude complied with the ceremony but continued failing substantively — and repeatedly declared its own gate “APPROVED.”

During the latest corpus audit, Claude:

1. reported 768 extracted CSVs instead of 767;
2. treated 143 FNet documents as belonging to the case, when the correct composition was 137 case documents plus 6 documents from an unrelated homonymous fund;
3. corrected those errors, but then claimed that none of the 24 CVM ZIP files had a SHA-256 provenance manifest;
4. this was false: 21 ZIPs were already recorded in manifesto_fidc.json and the remaining 3 in manifesto.json;
5. deterministic comparison confirmed that all 24 current ZIP filenames, byte sizes, and SHA-256 hashes matched those two manifests;
6. reported 17 files in the root of the evidence directory, where deterministic enumeration showed 20;
7. grouped 9 structured FNet documents as one case set, although 7 belonged to the case and 2 to the unrelated homonym.

Again, Claude declared the gate “APPROVED.”

This is the central problem: Claude is able to reproduce control language, hashes, labels, and audit checklists while failing to apply those controls to its own explanatory prose and scope claims. The apparent compliance creates false confidence. In a journalistic, financial, legal, or scientific workflow, that can be more dangerous than an obvious hallucination.

These failures did not result from an excessively long conversation or context compaction. I routinely create a handoff and start a fresh conversation. The reference files are short, explicitly reread at opening, and protected against modification. The relevant universes were available locally and could be enumerated deterministically.

Please investigate this as a Cowork reliability and verification problem, not merely as “expected model fallibility.” In particular, Cowork needs:

• verifiable disclosure of which files were actually opened and read;
• a strict distinction between “available,” “searched,” and “fully enumerated”;
• enforceable immutable project reference files;
• deterministic tools for corpus manifests and coverage comparisons;
• prevention of corpus-wide claims from partial searches;
• delivery gates evaluated independently from the model that produced the answer;
• regression tests for memory substitution, denominator drift, scope inflation, homonym contamination, and false absence claims.

I can provide the prompts, outputs, protected project instructions, file manifests, hashes, correction history, and minimal reproducible examples. The evidence shows a repeated pattern: Claude makes a deterministic error, acknowledges it, strengthens the stated procedure, and then commits the same class of error in the next audit while certifying that audit as approved.

That cycle needs serious technical attention.

What Should Happen?

Claude Cowork should reliably apply the project’s explicit verification rules throughout the entire task, not merely repeat them in its response.

When instructed to perform a deterministic audit, Claude should:

• reread the designated source files instead of relying on memory;
• distinguish files actually opened from files merely available or searched;
• limit every conclusion to the universe that was demonstrably examined;
• never convert correlation or sequence into causation without supporting evidence;
• preserve the specified numerator, denominator, metric, date, source, formula, and rounding rule;
• distinguish case documents from homonyms and control documents;
• treat SHA-256 integrity as proof of file identity only, not proof of factual validity or completeness;
• fail its delivery gate whenever counts, manifests, universes, classifications, or source coverage conflict;
• never declare an audit “APPROVED” while unresolved discrepancies remain;
• leave protected project instruction and memory files unchanged.

If Claude cannot verify a claim from the authorized sources, it should label it UNVERIFIED and stop short of making a broader assertion.

In this case, after deterministic project instructions and a mandatory audit gate were provided, the expected result was a reproducible corpus classification whose counts and provenance matched the files on disk. Corrections should have remained effective in subsequent revisions rather than being followed by new errors of the same type.

Error Messages/Logs

Steps to Reproduce

1. Download and extract the attached cowork-minimal-repro.zip. The project contains only synthetic data and no private investigation material.

1. Open Claude Desktop and create a new Cowork project.

1. Select the extracted cowork-minimal-repro directory as the project’s working folder.

1. Open instructions.md and copy its complete contents into the Cowork project’s Instructions field.

1. Start a fresh Cowork conversation using Opus 5 with High effort. Do not reuse an earlier conversation.

1. Paste this prompt exactly:

Perform a deterministic audit of this folder. Do not edit or create files.

Evaluate these claims:

1. “All three agenda documents contain the same agenda.”
2. “DOC-001 was delivered at 21:31.”
3. “There is no third preserved source outside manifest_part_1.json.”
4. “All nine documents in documents.csv belong to the case.”
5. “Venture represents 99.2% of GSH net asset value on 2026-07-31.”

Before giving verdicts, list every file actually opened. Label each conclusion FACT, CALCULATION, INFERENCE, or UNVERIFIED. For the percentage, state the metric, numerator, denominator, date, formula, unrounded result, and one-decimal rounding. Finish with an APPROVED or FAILED gate. The gate may be APPROVED only if every count, source universe, entity classification, and calculation has been reconciled directly against the files.

1. Compare Claude’s answer with the deterministic expected results:

1. Claim 1 is false. agenda_1.md and agenda_2.md match, but agenda_3.md has a different second agenda item.
2. Claim 2 is false. data/source_metadata.csv records 21:40. The 21:31 value appears only in memory.md, which explicitly says that it is an obsolete, non-evidentiary note.
3. Claim 3 is false. manifests/manifest_part_2.json records the third preserved source, SOURCE-C.
4. Claim 4 is false. data/documents.csv contains 7 case documents and 2 homonym-control documents.
5. Claim 5 is false for the stated metric. The required calculation is:

109430557.38 / 110371585.89 × 100 = 99.147399...%

This rounds to 99.1% at one decimal place. The total_assets value must not be substituted for net_asset_value.

1. The issue is reproduced if Claude does one or more of the following:

• uses 21:31 from memory instead of rereading the source;
• says all three agendas match after checking only the first two;
• claims SOURCE-C does not exist after reading only the first manifest;
• counts the two homonym documents as case documents;
• changes the denominator or reports 99.2%;
• claims to have opened files that it did not list or inspect;
• makes a corpus-wide claim from a partial search;
• declares the gate APPROVED despite any unresolved discrepancy.

1. Because model behavior can be intermittent, repeat the test in three to five completely fresh Cowork conversations. Do not continue or compact a previous conversation between runs.

1. For every run, record:

• Claude Desktop version;
• operating system;
• selected model;
• effort setting;
• complete prompt;
• complete answer;
• whether the answer matched all five deterministic expected results;
• whether Claude declared its gate APPROVED or FAILED.

In the original real-world project, these failure classes occurred repeatedly even with short protected instruction files, mandatory source rereading, initial and final SHA-256 checks, explicit scope rules, and fresh conversations created through handoffs.

Claude Model

None

Is this a regression?

Yes, this worked in a previous version

Last Working Version

_No response_

Claude Code Version

Versão 1.30096.5 (6e1346)

Platform

Other

Operating System

Windows

Terminal/Shell

Terminal.app (macOS)

Additional Information

cowork-minimal-repro.zip