[DOCS] WebFetch docs missing large-HTML truncation behavior

Status Fixed / completed
Maintainer reply ✓ Yes — bcherny
Activity 5 comments · opened Apr 22, 2026 · closed Aug 17, 2026
💡 Likely answer: A maintainer (bcherny, collaborator) responded on this thread — see the highlighted reply below.

Documentation Type

Missing documentation (feature not documented)

Documentation Location

https://code.claude.com/docs/en/tools-reference

Section/Topic

WebFetch tool behavior and limitations

Current Documentation

The docs currently say:

WebFetch — Fetches content from a specified URL

The Agent SDK references are similarly generic:

Fetches content from a URL and processes it with an AI model.

And the Python Agent SDK only documents the basic input/output shape:

{"url": str, "prompt": str} {"response": str, "url": str, "final_url": str | None, "status_code": int | None}

None of these pages mention that very large HTML pages may be truncated before HTML-to-markdown conversion.

What's Wrong or Missing?

Claude Code changelog v2.1.117 says:

Fixed WebFetch hanging on very large HTML pages by truncating input before HTML-to-markdown conversion

That change exposes user-visible behavior that is not documented anywhere in the Claude Code docs: when WebFetch receives a very large HTML page, Claude Code may truncate the raw HTML before converting it to markdown. Users who get incomplete fetch results from oversized pages currently have no official documentation explaining why content may be missing, that this is intentional, or that large monolithic HTML pages can behave differently from smaller pages.

Suggested Improvement

Add a short limitation note to the WebFetch documentation in the main tool reference, with matching notes in the Agent SDK references. The note should explain that:

  • very large HTML pages may be truncated before HTML-to-markdown conversion to prevent hangs
  • fetched output from oversized pages may therefore be partial
  • narrower, more targeted URLs are more reliable when users need complete page content

Impact

Medium - Makes feature difficult to understand

Additional Context

Affected Pages:

| Page | Context |
|------|---------|
| https://code.claude.com/docs/en/tools-reference | Main built-in tool reference for WebFetch |
| https://code.claude.com/docs/en/agent-sdk/typescript | TypeScript Agent SDK WebFetch input/output reference |
| https://code.claude.com/docs/en/agent-sdk/python | Python Agent SDK WebFetch input/output reference |
| https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool | Cross-reference: API web fetch docs already describe content limits/truncation for large fetched content |

Total scope: 4 pages affected

Source: Changelog v2.1.117

Exact changelog entry: Fixed WebFetch hanging on very large HTML pages by truncating input before HTML-to-markdown conversion

View original on GitHub ↗

5 Comments

coygeek · 3 months ago

Still relevant as of 2026-05-18. This documentation gap has not been addressed in the docs yet.

coygeek · 2 months ago

Verified against the latest docs; this still appears unresolved.

coygeek · 2 months ago

Verified against the latest docs; this still appears unresolved.

happy9zhang · 24 days ago

Additional data point (2026-08-06): the truncation is not limited to HTML and it's fully silent — which produces confident wrong answers from the summarizer model.

Repro through an agent embedding the Claude Code CLI:

  1. WebFetch on https://api.github.com/repos/makecindy/cindy/git/trees/main?recursive=1 — actual response is 2,498,762 bytes; the GitHub API itself reports "truncated": false with 7808 tree entries.
  2. Asked to list the repository's top-level directories → the tool answered "the repository has no packages/ directory" and listed only the alphabetically-first prefix of paths (.githooksapps/desktop/scripts/gen-help-kb.mjs, exactly up to the truncation point).
  3. curl-ing the same URL shows packages/ exists (28 packages) and the tree is complete.

The dangerous part is not the truncation itself but that it is silent: the content handed to the small summarizer model carries no "content was truncated" marker, so the model confidently asserts facts from a partial prefix. In agent workflows, confident misinformation is worse than an explicit error.

Beyond the docs gap, could the implementation also consider:

  • a truncation marker in the content passed to the summarizer,
  • fail-loud behavior when the cap is exceeded, and/or
  • skipping HTML→markdown conversion for non-HTML content types (JSON, etc.)?
bcherny collaborator · 14 days ago

Good news — the tools reference now has a dedicated "WebFetch tool behavior" section that documents this: large pages are truncated to a fixed character limit before processing, the extraction step is lossy by design, and for the unprocessed page you can use curl via Bash instead. See https://code.claude.com/docs/en/tools-reference#webfetch-tool-behavior. Closing as documented; feel free to open a new issue if the SDK reference pages still need a matching note.

🤖 Generated with Claude Code