Feature Request: Enhance Web Search to Utilize `llms.txt` for Better Context

Title: Feature Request: Enhance Web Search to Utilize llms.txt for Better Context

Labels: enhancement, feature-request, tools, web-search

Is your feature request related to a problem? Please describe.

Currently, when Claude Code performs a web search or fetches content from a URL, it relies on standard page scraping and search engine results. This process can sometimes miss a high-signal, developer-curated source of information that is becoming more common: the llms.txt file.

Many tech companies (e.g., Stripe) now host an llms.txt file at the root of their domain to provide a concise, structured summary of their site's purpose, key links, and APIs specifically for AI agents. This is a powerful but underutilized "best kept secret" that Claude Code currently overlooks.

For example, https://stripe.com/llms.txt provides an excellent overview for an agent, but this file is not typically linked from the homepage and must be discovered proactively.

Describe the solution you'd like

I propose that the WebSearch and WebFetch tools be updated with the following logic:

1. When a user provides a URL or a search query that resolves to a root domain (e.g., stripe.com), the tool should automatically and concurrently attempt to fetch https://<domain>/llms.txt.
2. If an llms.txt file is found (i.e., returns a 200 OK status), its content should be prioritized and heavily weighted as the primary source of context for understanding the website.
3. If the file does not exist (e.g., a 404 Not Found), the tool should proceed with its standard search/fetch behavior without any user-facing error. The process should be seamless.

Describe alternatives you've considered

The alternative is the current approach, which relies solely on web scraping and search engine indexing. This is less efficient and may not capture the most important, up-to-date information as intended by the site's owners for AI interaction. Manually asking Claude to check for the file is cumbersome and not scalable.

Additional context

Adopting this standard would give Claude Code a significant advantage in accurately and efficiently understanding complex websites and their APIs. It aligns with the "Unix philosophy" of using simple, standardized text files for communication and would improve the quality of responses for a wide range of tasks, from codebase questions involving third-party APIs to general web research. This small change could dramatically improve the quality of context Claude receives from many websites.