WebFetch passes raw PDF bytes to its summarizer with no text extraction and no "unreadable content" signal (still in 2.1.250)

Filing fresh because #23694 — "WebFetch should extract text from PDFs before passing to the summarizer model" — is locked and closed as not_planned (2026-04-04), so it cannot take a comment. The behaviour is still present in 2.1.250.

The deterministic part

WebFetch on a URL that returns a PDF passes the raw response bytes to its summarizer model. Across 16 controlled trials (4 public PDFs × 4 prompt styles, 242KB–2.7MB) the result was identical every time:

• the [Binary content (application/pdf, <size>) also saved to <path>] suffix appeared 16/16
• text extraction succeeded 0/16 — the summarizer never received document prose, only bytes

That suffix is concatenated after the model's answer, so the summarizer itself gets no indication the payload is unreadable. Whether the caller receives an honest refusal depends entirely on the model recognising %PDF-1.5 / FlateDecode byte soup on its own.

The consequence: real, but uncommon

Usually it does recognise it — 16/16 refused, correctly describing the content as binary or compressed stream data. But I have one captured instance where it did not.

A subagent called:

WebFetch(url: "https://www.site.uottawa.ca/~stan/csi5387/NELL-notes.pdf",
         prompt: "Quote VERBATIM the sentences describing how candidate facts are PROMOTED to beliefs ...")

and received a confident answer containing five bolded "verbatim" quotes, each with a section attribution, e.g.:

The paper explicitly discusses human involvement: "Human reviewers examine promoted beliefs that fall into uncertain confidence ranges to validate system decisions" (Section 4, Learning from Human Feedback).

None of the five strings exist in the fetched PDF, nor in the paper it summarises (Carlson et al., AAAI 2010). The cited sections do not exist either — neither document has numbered subsections, and neither contains "Learning from Human Feedback". Longest contiguous match for any of the five quotes is four tokens. Verified twice, independently, by extracting both PDFs with a real PDF text extractor and searching case- and whitespace-insensitively. Where the real paper is checkable it says something different: promotion is posterior > 0.9 from a single source, or lower-confidence candidates proposed by multiple sources.

Rate, stated honestly: 1 fabrication in 21 total observations (~5%; 95% upper bound ≈ 17%). This is not a claim that it always fabricates — it is that nothing on this path prevents it, and it demonstrably happens. The failure mode matters more than the frequency: the caller receives quotation marks and a section number, which is what sourced evidence looks like.

The contrast that makes it fixable

A 404 on a different PDF URL in the same session returned exactly the right shape:

The server returned HTTP 404 Not Found. The response body was not retrieved.

So the tool reports inability cleanly when the fetch fails. It is the successful fetch of an unconvertible binary that has no such path.

Suggested fix, cheapest first

Tell the summarizer — not only the caller, after the answer — that the content could not be converted to text, and point at the saved file. PDF text extraction, the original request in #23694, would be better still.

Related open issues look like the same failure on non-PDF inputs: #87470 (a 500 error page summarised as a live job posting) and #86581 (stale training-data content spliced into a real retrieval). The documentation gap is #30942 (also closed as stale): the tools reference documents PDF support for Read but says nothing about WebFetch and binary content.

Environment

Claude Code 2.1.250, Linux (WSL2).