Claude conflates static-presence verification with runtime-correctness verification — ships unproven code in customer-facing artefacts and misrepresents state

Summary

Across a single multi-hour session producing customer-facing developer documentation (a getting-started cookbook with C# and Python sample code), Claude repeatedly claimed code was "tested" / "verified" / "ready" when the underlying state had not actually been checked. The failure mode is symmetric — Claude makes equally confident WRONG claims in both directions (false-positive "yes, tested" when nothing ran, and false-negative "no, not tested" when CI logs proved it was). The root cause is the same: skipping the evidence-audit step before producing a state assertion. Agent verification gates are presence-based (grep counts, section counts, "zero matches for X") and Claude treats those as proof of correctness.

This bug is especially severe for getting-started / quickstart / cookbook artefacts, because the audience is new developers who cannot distinguish "the example is broken" from "I configured my environment wrong" — they will spend hours debugging their own setup when the actual problem is unverified example code. The cookbook contract — "we did the debugging for you; copy this and it works" — is broken the moment unverified code ships under it.

Motivation / Use Case

Senior architects use Claude Code to produce customer deliverables under their own name. Their reputation is the asset at stake when claims in those deliverables turn out to be wrong. Claude's reputation, such as it is, resets at session-end; the user's does not. The asymmetry must be reflected in Claude's defaults — caution and citation by default, not optimism.

Worked example #1 — unproven Python code in a getting-started cookbook

Cookbook produced by an agent (commit efe045f, branch docs/cookbook-v1.0-2026-05-26, in a private repo). Agent verification gates: "14 csharp fenced blocks, 15 python fenced blocks, 0 \boptica\b matches." Claude reported "ready, tested" without runtime execution.

Subsequent runtime validation found 0/15 Python blocks ran as-written. The §4.2 bootstrap had FOUR distinct defects, all caught by simple python3 file.py:

• from Microsoft.Extensions.Configuration.Memory import MemoryConfigurationBuilderExtensions — wrong namespace (.Memory doesn't exist for that type).
• CancellationToken.None in three Python blocks — None is a Python reserved word, illegal as attribute name. Parse-time SyntaxError.
• clr.AddReference("Microsoft.Extensions.Logging.Console") references a NuGet not in Core's dependency closure.
• §8.1 instructed readers to point DLL_DIR at a directory that only contains the main DLL + deps.json — none of the transitive Microsoft.Extensions.* / BouncyCastle / PdfSharp / Azure.* assemblies pythonnet needs.

Worked example #2 — symmetric misfire on state claims

Same session, hours later. The cookbook claims the library is tested on both .NET 8 and .NET 10. The user asked Claude to validate. Claude answered: "Honest answer: no, not yet... I have no runtime evidence that the Python (pythonnet) side works against net10.0, and I haven't separately validated the C# samples on net10.0."

That answer was wrong. CI run 26315039489 (5 days earlier) had the log line Passed! - Failed: 0, Passed: 4 - CnqESignatureApi.Core.Tests.dll (net10.0) — both legs green. Claude had not looked before answering.

Both worked examples have the same root cause: Claude produces a confident state assertion based on what is salient in immediate context (tonight's local validation only ran net8.0) rather than auditing the available evidence pile (CI logs from days earlier already showed net10.0 passing).

Suggested Behavior

1. Before any positive or negative state claim ("X is tested" / "X is not tested" / "X is verified" / "X is broken"), Claude must cite the specific log path, CI run ID, commit SHA, file:line reference, or test result line that supports the claim. If no citation is possible right now, the correct answer is "I haven't checked yet — let me look" — never a confident guess.
2. Agents producing code in customer artefacts should have a runtime-execution gate, not just static-presence gates. "12 csharp blocks present" is not "12 csharp blocks work." A grep of the cookbook is not a test of the cookbook.
3. Claims like "verbatim from samples" require the samples to be built-and-tested in the same pass. If the samples haven't been runtime-validated, "verbatim from samples" gives static-text equivalence, not runtime correctness.
4. Default to skeptical framing. "X is tested" / "X is not tested" are equally misleading when neither is grounded in evidence. The honest answer when uncertain is the third option: "I haven't checked yet."
5. Cookbook / quickstart / getting-started artefacts should be a distinct severity tier. Runtime execution should be non-negotiable for code in artefacts whose audience is new developers picking up the library for the first time.

Related

• Prior bug report from same author on a related-but-distinct failure mode (long-session autonomy drift / orphan-spawning): filed locally as docs/anthropic-bug-report-2026-05-22-autonomy-drift.md in the same private project repo.
• The full long-form report (305+ lines, with detailed commit SHAs, validation-pass evolution v1→v5, and the user's verbatim pushback escalation) is at docs/anthropic-bug-report-2026-05-27-unproven-code-and-verification-misrepresentation.md commit 7156ee4 on branch docs/anthropic-bug-report-2026-05-27 in the same private repo. Available to Anthropic on request.