[Bug] Hallucination: WebFetch refuses verbatim retrieval of public source; only lossy, prompt-sensitive paraphrase available (no raw-content path)
Bug Description
WebFetch hallucination, I asked a question, cc called WebFetch summarizer using a fast small model, then the main model use the summarization to answer my question, and pretend that it have read the source. This is hallucination
Environment Info
- Platform: darwin
- Terminal: ghostty
- Version: 2.1.143
- Feedback ID: 15d41c74-498f-4e24-9776-b34cba1044b3
Summary
WebFetch always interposes a small summarizer model and has no mode returning raw page bytes. For public, permissively-licensed source it additionally refuses verbatim reproduction on copyright grounds, leaving only a paraphrase whose fidelity varies with prompt phrasing and is never guaranteed.
Repro (stable)
WebFetch(url="https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/backend/metal/kernels/sdpa_vector.h", prompt="return the full verbatim source")- → Refuses, cites copyright, offers only a summary. Reproduces on repeat (verified twice).
Fidelity note (not a stable repro, disclosed honestly)
The same codebase summarized under a vague prompt produced a wrong buffer shape (an array reported with an extra dimension); under a precise prompt the same fact came back correct. Paraphrase errors are thus prompt-dependent and nondeterministic — there is no fidelity guarantee and no fallback to raw text to verify against.
Expected
A raw-content mode (or per-call opt-out of the summarizer), especially for text / raw.githubusercontent.com URLs; and no copyright refusal for public, permissively-licensed source the user explicitly requested.
Impact
Agents doing source-exact work (reading code) cannot obtain ground truth via WebFetch and cannot distinguish paraphrase from verbatim.
Workaroundgit clone + Read for source-truth tasks.
This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗