[FEATURE] Proactively fetch llms.txt and Claude.md from site root when reading documentation URLs

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Description:

When a user asks Claude Code to read a page from an external website, Claude should automatically attempt to fetch AI hint files from the site root before fetching the target
page:

• <scheme>://<host>/llms.txt
• <scheme>://<host>/Claude.md

Problem:

Many documentation sites use client-side rendering frameworks (such as Redocusaurus or React-based portals) that return empty HTML shells when
fetched directly. Site maintainers already invest time in publishing llms.txt and Claude.md files at the site root with AI-specific instructions — such as fallback URLs to raw
OpenAPI specs — precisely to handle this. However, Claude never checks for these files by default, making them effectively useless unless the user explicitly knows to point
Claude at them.

Proposed Solution

Expected Behavior:

When given a URL to read, Claude should:

1. Fetch <site-root>/llms.txt and <site-root>/Claude.md in parallel (ignoring 404s silently)
2. Use any instructions found there to determine the correct way to retrieve the actual content
3. Proceed with fetching the target page using that guidance

Why This Matters:

llms.txt is an emerging standard designed specifically for this purpose — analogous to how robots.txt guides crawlers. Site owners are already doing their part by publishing
these files. Claude adopting this as default behavior would close the loop and make the standard actually useful, without requiring users to know about or manually invoke it.

Impact: Low overhead (at most 2 extra HTTP requests, both parallelizable), high value for documentation-heavy workflows.

Alternative Solutions

_No response_

Priority

High - Significant impact on productivity

Feature Category

CLI commands and flags

Use Case Example

_No response_

Additional Context

_No response_