[DOCS] Document that WebFetch identifies as `Claude-User` so site operators can allow or block it via `robots.txt`

Documentation Type

Missing documentation (feature not documented)

Documentation Location

https://code.claude.com/docs/en/tools-reference

Section/Topic

The WebFetch row in the tools reference table and/or a dedicated section on WebFetch network behavior.

Current Documentation

The tools-reference page describes WebFetch only as:

Fetches content from a specified URL

No mention is made of the HTTP user-agent string sent by the tool, how site operators can identify Claude Code traffic, or how robots.txt directives apply to WebFetch requests.

The support article at support.claude.com ("Does Anthropic crawl data from the web, and how can site owners block the crawler?") describes the Claude-User agent in general terms for Anthropic's crawlers, but does not connect it specifically to Claude Code's WebFetch tool. The code.claude.com docs make no reference to Claude-User, user-agent strings, or robots.txt at all.

What's Wrong or Missing?

As of Claude Code v2.1.83, WebFetch now identifies itself as Claude-User in the HTTP User-Agent header. This is a meaningful, user-facing behavior change for two audiences:

1. Developers using Claude Code – need to know that their WebFetch-driven requests are identifiable on the server side, which affects debugging, analytics, and access control.
2. Site operators – need to know that they can selectively allow or block Claude Code's fetch traffic using Claude-User in robots.txt, the same mechanism Anthropic documents for its other bots.

Neither the tools-reference page nor any other code.claude.com page documents this behavior. A developer who wants to know how WebFetch presents itself to servers, or a site owner who wants to allow Claude Code through their robots.txt, has no documented path from the Claude Code docs to discover this.

Suggested Improvement

Add a "Network identity" or "HTTP behavior" note to the WebFetch entry in the tools-reference page (and/or to a dedicated WebFetch documentation page if one is created). For example:

User-Agent: WebFetch identifies itself to web servers as Claude-User. Site operators can use this user-agent string in robots.txt to allow or disallow Claude Code fetch requests. See Anthropic's bot documentation for details on configuring per-bot access.

This connects the Claude Code tool documentation to the existing support article, giving both developers and site operators a clear path to the relevant information.

Affected Pages:

• https://code.claude.com/docs/en/tools-reference (primary — add user-agent note to WebFetch row or in a new subsection)

Impact

Medium - Makes feature difficult to understand

Additional Context

Source: Changelog v2.1.83 — "Improved WebFetch to identify as Claude-User so site operators can recognize and allowlist Claude Code traffic via robots.txt"

The Claude-User bot is already described in the Anthropic support article at https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, which explains that Claude-User "supports Claude AI users" and that "disabling Claude-User on your site prevents our system from retrieving your content in response to a user query." The Claude Code docs should explicitly connect WebFetch to this existing bot documentation so the relationship is clear to all parties.