[Bug] Hallucination: WebFetch refuses verbatim retrieval of public source; only lossy, prompt-sensitive paraphrase available (no raw-content path)

Resolved 💬 4 comments Opened May 17, 2026 by WindChimeRan Closed Jun 16, 2026

Bug Description
WebFetch hallucination, I asked a question, cc called WebFetch summarizer using a fast small model, then the main model use the summarization to answer my question, and pretend that it have read the source. This is hallucination

Environment Info

  • Platform: darwin
  • Terminal: ghostty
  • Version: 2.1.143
  • Feedback ID: 15d41c74-498f-4e24-9776-b34cba1044b3

Summary
WebFetch always interposes a small summarizer model and has no mode returning raw page bytes. For public, permissively-licensed source it additionally refuses verbatim reproduction on copyright grounds, leaving only a paraphrase whose fidelity varies with prompt phrasing and is never guaranteed.

Repro (stable)

  1. WebFetch(url="https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/backend/metal/kernels/sdpa_vector.h", prompt="return the full verbatim source")
  2. → Refuses, cites copyright, offers only a summary. Reproduces on repeat (verified twice).

Fidelity note (not a stable repro, disclosed honestly)
The same codebase summarized under a vague prompt produced a wrong buffer shape (an array reported with an extra dimension); under a precise prompt the same fact came back correct. Paraphrase errors are thus prompt-dependent and nondeterministic — there is no fidelity guarantee and no fallback to raw text to verify against.

Expected
A raw-content mode (or per-call opt-out of the summarizer), especially for text / raw.githubusercontent.com URLs; and no copyright refusal for public, permissively-licensed source the user explicitly requested.

Impact
Agents doing source-exact work (reading code) cannot obtain ground truth via WebFetch and cannot distinguish paraphrase from verbatim.

Workaround
git clone + Read for source-truth tasks.

View original on GitHub ↗

This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗