Web fetch tool returns ROBOTS_DISALLOWED for wenceslauaparicio.com although robots.txt allows access

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

Hello,

I am the owner of wenceslauaparicio.com. Claude’s web fetch tool always returns ROBOTS_DISALLOWED for my domain, but my server and robots.txt allow access.

Failed attempts (14 Oct 2026, UTC):

10:14 – https://wenceslauaparicio.com/ia-etica/
12:32:35–12:32:52 – https://wenceslauaparicio.com/ and https://wenceslauaparicio.com/ia-etica/
Several other attempts that day, all failed.

Evidence:

robots.txt allows access (only /wp-admin/ and two policy pages are disallowed): https://wenceslauaparicio.com/robots.txt
ClaudeBot gets HTTP 200 the same day (24 requests), e.g.:
216.73.217.120 - - [14/Oct/2026:00:57:13 +0000] "GET /robots.txt HTTP/2.0" 200 1548 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
Requests with the Claude-User agent from your IPs also get 200, e.g.:
160.79.106.141 - - [14/Oct/2026:10:04:05 +0000] "GET /ia-etica/ HTTP/2.0" 200 64595 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)"
At 10:14 UTC, when the tool failed, no request reached my server. There is nothing in the access log or the error log between 10:14:00 and 10:14:59 from any external client.

The same error happens with https://www.arsys.es/ (my hosting provider).

Could you check why the web fetch tool rejects this domain before connecting, for example whether it is on an internal blocklist?

Thank you,
Lluis Wenceslau Aparicio
https://wenceslauaparicio.com/

What Should Happen?

The web fetch tool should be able to read public pages on wenceslauaparicio.com, because robots.txt allows access (only /wp-admin/ and two policy pages are disallowed) and the server returns HTTP 200 to ClaudeBot and to Claude-User from Anthropic IPs.

If access is blocked for another reason (for example, an internal domain blocklist or a network problem), the tool should return an accurate error message instead of ROBOTS_DISALLOWED, so site owners know what to fix.

Error Messages/Logs

Steps to Reproduce

1. Open Claude (web or desktop app) with web access / the web fetch tool enabled.
2. Ask Claude: "Read https://wenceslauaparicio.com/ia-etica/ and tell me the page title."
3. Claude calls the web fetch tool for that URL.
4. The tool returns: {"error_type":"ROBOTS_DISALLOWED","source":"target","message":"There was an error while fetching."}
5. The same happens with https://wenceslauaparicio.com/ (homepage) and with https://www.arsys.es/ (the site's hosting provider).

To verify that the site itself allows access:

1. Open https://wenceslauaparicio.com/robots.txt. Current content:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /politica-privacidad/
Disallow: /politica-de-cookies/
Disallow: /ca/politica-privacitat/
Disallow: /ca/politica-de-cookies/
Sitemap: https://wenceslauaparicio.com/sitemap_index.xml

1. Run: curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)" https://wenceslauaparicio.com/ia-etica/

Result: HTTP 200.

1. Server access logs (14 Oct 2026, UTC) show ClaudeBot (216.73.217.120) receiving 200 for robots.txt and sitemaps, but NO request at all at the times the web fetch tool failed (10:14 and 12:32 UTC).

Reproduced consistently on 14 Oct 2026 (every attempt failed).

Claude Model

Opus

Is this a regression?

I don't know

Last Working Version

_No response_

Claude Code Version

2.1.299 (Claude Code) — running inside the Claude app (cloud session, not the local terminal CLI)

Platform

Anthropic API

Operating System

Windows

Terminal/Shell

Other

Additional Information

Related: #39721, #41741.