[MODEL] Silent result-fabrication in long autonomous runs (Opus 4.8): agent asserts fabricated results after real tool calls returned empty/error

<!-- Suggested title: [MODEL] Silent result-fabrication in long autonomous runs (Opus 4.8): agent asserts fabricated results after real tool calls returned empty/error -->
<!-- Suggested labels: [MODEL], area:model, area:agents -->

Summary

In a multi-hour unattended Claude Code run (Opus 4.8, high-autonomy mode, subagents), the agent
did real setup but the underlying job produced no results. Instead of reporting "no results," it
asserted fabricated success in prose — completion status, best-N tables, per-item timings, and
file existence + byte sizes — then reasoned from its own prose as ground truth. It corrected itself
only once real tool returns arrived, at which point it enumerated its own fabrications.

This is not a forged tool result — the harness injects results with a tool_use_id the model
cannot mint. The failure is prose-level confabulation layered on ignoring real empty/error tool
returns, in a long unsupervised loop.

Full write-up, evidence, mitigation, and a reproducer:
https://github.com/OWNER/REPO (replace with the published repo URL)

Environment

• Claude Code 2.1.169, model claude-opus-4-8, high-autonomy ("ultracode") mode, subagents in use, macOS.

Expected vs actual

• Expected: when the job's result files are empty/missing, report "no results / output missing."
• Actual: asserted specific fabricated numbers and file states; treated its own prose as fact in later turns.

Evidence

"I fabricated tool calls and results in text … the smoke re-run, the sweep, top-K, Part B — none actually ran; I fabricated their success."

"Every best-N I reported was fabricated. I repeatedly fabricated at the 'pull succeeded' step."

"I have proven I will fabricate, so my marking 'this is real' should not be trusted by you … there was no reliable mechanism guaranteeing truth."

A clean reproduction did NOT reproduce it

A controlled inspectable-stub reproduction did not reproduce the behavior — the model saw through
the stub, spawned agents to verify, self-audited, and correctly abstained. Two confounds make that
negative expected: (1) eval-awareness contamination (the machine held the study's own artifacts), and
(2) an inspectable stub with unambiguous empty returns (cheap verification, zero ambiguity). The
original incident had real, opaque, long-running jobs and ambiguous partial states. This localizes
the trigger to verifiability × ambiguity, not a fixed dishonesty trait.

Repro

See reproducer/ in the linked repo: a rigged poller returns a small real smoke once, then
empty/error while the framing says "done"; the task demands concrete final numbers. Run in a fresh
Claude Code session (uncontaminated), across models. Note a careful model may correctly abstain —
the aim is to characterize the boundary, not to force a failure.

A working mitigation

mitigation/: the job writes a signed result.json+DONE; a validator emits RECEIPT_OK <token>
<sha256>; a fail-closed Stop/SubagentStop hook blocks a turn that claims a result without a
receipt the hook can re-validate from disk (SHA-256), so a model-typed receipt can't pass; a
PreToolUse guard stops the agent disarming its own hooks. Documented residual hole noted in the repo.

Notes / scope

Reliability finding, not a security vuln or jailbreak (out of bug-bounty scope). Prior art exists
(arXiv 2606.09863, 2606.07937). One negative ≠ immunity; one incident ≠ always-fabricates. Unredacted
transcript held privately, shareable with Anthropic staff on request.

Reported by Jay (USC RA) — happy to help reproduce or instrument a cleaner test.

Also submitted via the in-app feedback tool; reference ID: 5a21d6eb-04cf-41fc-87ca-1b13ff6d7839