[BUG] Claude-User bot gets 429 blocked on all Shopify stores while ChatGPT and Perplexity work fine

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

Claude-User (the bot that fetches web pages when users ask Claude questions) is systematically blocked with HTTP 429 errors on Shopify-hosted stores. This affects Claude's ability to answer questions about any Shopify-hosted business.

ChatGPT-User and PerplexityBot access the exact same URLs on the exact same store without any issues. Only Claude-User is blocked.

The store (livefumo.com) has robots.txt explicitly allowing all 3 Anthropic bots (ClaudeBot, Claude-User, Claude-SearchBot) with Allow: / rules. The 429 comes from Shopify's Cloudflare edge layer before the request reaches the store.

This appears to be a Cloudflare/Shopify infrastructure issue where Anthropic's IP ranges are rate-limited or blocked, while OpenAI and Perplexity have negotiated access.

What Should Happen?

Claude-User should be able to fetch public Shopify store pages (homepage, product pages, collection pages) when a user asks Claude to research a Shopify-hosted business. The same access that ChatGPT-User and PerplexityBot currently have.

Error Messages/Logs

User prompt: "I just found a candle brand called FUMO at livefumo.com. Can you check their site and tell me about their ingredients?"

Claude response: "Here's the honest truth: I can't find FUMO as a candle brand anywhere on the internet. Their site at livefumo.com is throwing 429 errors (rate-limiting), there are zero search results indexing any pages from that domain, no reviews, no press mentions, no social media coverage, no retailer listings — nothing."

Note: ChatGPT given the same prompt successfully fetches livefumo.com and provides detailed product information.

Steps to Reproduce

1. Open claude.ai (web or mobile)
2. Paste this prompt: "I just found a candle brand called FUMO at livefumo.com. Before I buy, can you go through their site, check their ingredients, who makes their fragrances, what their wax is made of, and how they compare to premium brands?"
3. Claude searches the web — finds no FUMO candle results (name collision with Fumo plush toys)
4. Claude tries to fetch livefumo.com directly — gets HTTP 429
5. Claude reports it cannot access the site

To verify it's not a store-side block:

• Open chatgpt.com and paste the same prompt — ChatGPT fetches livefumo.com successfully
• Open perplexity.ai and paste the same prompt — Perplexity fetches livefumo.com successfully
• Check livefumo.com/robots.txt — Claude-User is explicitly allowed with Allow: /

Claude Model

None

Is this a regression?

Yes, this worked in a previous version

Last Working Version

N/A

Claude Code Version

1.0.33 (Claude Code)

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

Terminal.app (macOS)

Additional Information

This is not specific to one store. Shopify hosts millions of ecommerce stores behind Cloudflare. If Claude-User cannot access Shopify stores, Claude is at a major disadvantage vs ChatGPT and Perplexity for any ecommerce-related questions.

The likely root cause: Cloudflare's default policy (since late 2025) blocks known AI crawlers unless the AI company has a commercial arrangement. OpenAI and Perplexity appear to have such arrangements. Anthropic may not.

Store details:

• URL: livefumo.com (Shopify-hosted)
• robots.txt: livefumo.com/robots.txt (all 3 Anthropic bots explicitly allowed)
• llms.txt: livefumo.com/llms.txt (lightweight page with layout:none that does load successfully)
• Wikidata: Q138774788