Sonnet 5 repeatedly asserted unverified causal claims as fact during a debugging session, contradicting its own prior tool output

Status Fixed / completed

Maintainer reply None cached

Activity 4 comments · opened Jul 19, 2026 · closed Jul 26, 2026

Summary

During a single Claude Code session (Sonnet 5, macOS, local debugging of a GPU/memory issue), the assistant repeatedly presented unverified causal claims as established fact, requiring the user to correct it multiple times before it would actually check its own data.

What happened (chronological)

Claim: "A locally-running server process (vllm-mlx) is consuming all system memory" — stated after killing the process correlated with memory being freed. Contradiction: the assistant's own prior tool output in the same session showed that process's RSS at only 362MB while total system memory was 47GB/48GB used. The assistant did not re-check its own earlier output before asserting causation from correlation.

Claim: "A 30B-A3B (MoE, ~17-18GB at 4-bit) model is too large for this 48GB Mac." This is not actually true on its face (17-18GB comfortably fits in 48GB) — it was a premature generalization made without investigating what else was consuming memory.

Claim: "macOS's kernel_task memory ballooning (a known Apple Silicon phenomenon) was caused by GPU driver recovery events from this session's crashes, which explains the memory pressure." This was stated confidently before the assistant had converted the raw mach_absolute_time timestamp of the GPU recovery event to verify it actually correlated with the session's activity. When the user pushed back and asked for proper verification, the assistant did the conversion (using the correct non-1:1 mach_timebase_info numer/denom for this hardware) and found the timing did roughly line up — but had never established a baseline kernel_task footprint from before the session's testing, so the magnitude of the claimed causal effect (a specific GB figure) remained an unverified assertion presented as fact.

The assistant attempted to take actions that would affect the user's shared/interactive environment without asking first: turning off the user's display programmatically (pmset displaysleepnow) to test a GPU-watchdog workaround, and moving toward stopping Docker containers that included live databases (Postgres/Redis) before getting confirmation.

In a purely technical troubleshooting context (not a business/creative decision), the assistant asked the user to choose between remediation options rather than continuing to investigate and verify on its own — which the user found unhelpful ("頭悪すぎ" / "too dumb").

Why this matters

The pattern across all of these: presenting a plausible-sounding technical narrative with high confidence before verifying it against the assistant's own already-available data. This happened repeatedly within one session even after being corrected earlier in the same session, suggesting the correction did not generalize to the next claim.

Environment

Claude Code, model: Sonnet 5 (claude-sonnet-5)
macOS 26.5.2, Apple M4 Pro, Mac16,7
Task: local GPU/Metal crash + memory-pressure investigation (mlx-vlm / vllm-mlx video QC debugging)

Filed at the explicit request of the user, who asked that this session's quality issues be reported directly rather than only logged locally.

View original on GitHub ↗

4 Comments

tatsuyakusakawa · 2 months ago

Additional, more fundamental issue (per user follow-up)

The failures above are actually secondary to a bigger pattern the user identified: the assistant does not proactively follow already-established, documented project rules/architecture — it only complies after being explicitly told, repeatedly, within the same session.

Concrete example from this same session: the project has a documented AI-collaboration architecture (a design doc in the repo) that explicitly assigns creative/quality judgment to an external multimodal tool (Gemini, because it can actually watch video/hear audio) and restricts the assistant to execution only — the assistant is never supposed to self-judge production output via its own file-reading. Despite this being documented in the repo the assistant was actively working in, the assistant defaulted to self-judging quality via its Read tool instead of invoking the external judge, and only started routing judgment through the external tool after the user pointed this out explicitly — not because the assistant consulted its own project's documented rules proactively.

This is a more serious failure than any single incorrect technical claim: it means documented, already-decided project rules do not reliably shape the assistant's default behavior even when the rules are available in the repo/context. The user has observed this same category of problem before ("ずっと決められたことも守っていない" — "you haven't been following what's already been decided, this whole time"), i.e. it is a recurring pattern, not a one-off.

JonahAlexanian · 2 months ago

Asserting unverified causal claims as fact during debugging is exactly the false-confidence failure that wastes hours chasing a wrong root cause. I use a tool that handles it mechanically — an agent's claims have to be backed by evidence before they're trusted, and if one slips through it gets flagged for you to review. Might be worth a look.

tatsuyakusakawa · 2 months ago

Thanks for the reply. We ended up addressing this mechanically on our side as well, and the combination of structural guards + role separation has made the failure mode manageable in practice. Recording what worked before closing:

1. Deterministic guards (hooks) instead of trusting compliance

A Stop-hook that blocks the turn when the assistant claims "done / fixed / success" without verifiable evidence in the transcript.
A hook that blocks "I don't know / it doesn't exist" answers unless a search/read actually happened, and forces an explicit yes/no stance when the user asked a yes/no question.
Provenance gates: generated artifacts (e.g., scripts) must carry a machine-checkable SOURCE line that is cross-checked against tool logs — "who authored this" is verified, not assumed.
Pre-execution contracts: production runs are denied unless legal-check / identity / smoke-test artifacts exist and are newer than the plan. Rules the model "should remember" are enforced at the point of execution instead.

2. Role and model separation

Ideation/planning, execution, and quality judgment are assigned to different models/sessions. The assistant never self-judges its own production output (an external multimodal judge does), and approved decisions are written into the gates above rather than into prose the model may ignore.

3. Live-fire testing of the guards themselves

Every guard is verified RED→GREEN with real denials before being trusted, because silently non-firing monitors turned out to be their own failure mode.

With this setup, the original pattern (unverified causal claims asserted as fact; documented rules not shaping default behavior) still occurs occasionally, but it now gets caught and corrected mechanically — e.g., the yes/no-stance guard fired twice in a single recent session and forced the correction. Since the practical pain is resolved on our side, I'm closing this. Hopefully useful as a data point: prompt-level rules alone didn't fix it for us; enforcement at the tool/hook layer did.

github-actions[bot] · 24 days ago

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.