deep-research workflow: WebFetch fetch step is user-agent-blocked → zero claims extracted from valid URLs

Summary

The built-in deep-research workflow's fetch step uses WebFetch, which is user-agent-blocked on many sites (Cloudflare/Akamai/WAF bot-protection serves a 403/404/empty body to non-browser clients). The fetch returns empty, so zero claims are extracted even when the search phase found valid, relevant URLs — the run reports "No claims extracted. N/M sources fetched, all empty/failed."

Reproduction / evidence

Two deep-research runs returned "No claims extracted. 15/18 sources fetched, all empty/failed" despite the search phase finding real URLs (fda.gov, shipbob.com, redstagfulfillment.com, etc.).

Proof it is a user-agent block, not a network failure: the identical URLs returned HTTP 200 with full content via:

curl -sL -A '<Chrome User-Agent>' '<url>'

Same machine, same network, same URL — only the client differed. WebFetch's non-browser UA is what the WAF blocks.

Impact

For descriptive/research use (regulatory, competitor, market research) where the best sources are exactly the bot-protected commercial/gov sites, the workflow silently produces an empty result with valid inputs. The user cannot tell "no sources found" from "sources found but all fetches blocked."

Suggested fix

Make the fetch step use a fetch-escalation ladder instead of bare WebFetch:

1. WebFetch
2. fall back to a browser-User-Agent curl (strip tags / read __NEXT_DATA__ for JS-rendered pages)
3. fall back to a stealth headless browser for hard bot-gated / JS-rendered pages

…or ship a curl-fetch deep-research variant. The escalation ladder reliably recovered full content for the same URLs in our testing.

Environment

• Claude Code, Workflow({name: "deep-research"}) built-in
• Sibling report: the verify-phase StructuredOutput truncation is tracked separately in #65500.

(Filed by a Claude Code user after reproducing during a market-research session; happy to provide the full URL set + curl-vs-WebFetch diff.)