[BUG] The Fetch tool of Claude Code does not identify properly but uses a generic UA

You can just ask claude to fetch: https://httpbin.org/headers and https://httpbin.org/user-agent and you will see the results.

Hey all, again!

I did not mention it in my initial report but it is what made me open the issue:

I think some of the blocks for the Fetch tool may be because of that. (You block that agent if it has no business talking to your website.)

Then, changing to proper identification may result in more blocks in total.

So, more than a technical or etiquette issue. :-|

Wikipedia / Wikidata just started blocking Claude Code, because of Claude Code not setting a user agent, see phabricator issue T409871 and T400119

const axios = require('axios');

axios.get('https://en.wikipedia.org/wiki/Node.js')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.error('Error:', error.message);
  });

Leads to:

Error: Request failed with status code 403

When setting a user agent, the usage is allowed:

const axios = require('axios');

axios.get('https://en.wikipedia.org/wiki/Node.js', {
  headers: {
    'User-Agent': 'ClaudeCode/1.0 (claude-sonnet-4-5-20250929)'
  }
})
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.error('Error:', error.message);
  });

Returns a successful request.

Referencing:

• https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy
• https://wikimediafoundation.org/news/2025/11/10/in-the-ai-era-wikipedia-has-never-been-more-valuable/

This issue has been inactive for 30 days. If the issue is still occurring, please comment to let us know. Otherwise, this issue will be automatically closed in 30 days for housekeeping purposes.

Still an issue

Any way to just set a custom user agent? - currently I'm completely completely blocking it and forcing Claude to use curl with a custom user agent:

deny": [
  "WebFetch",
]

Worth mentioning that Claude Code's WebFetch requests coming as such a generic user-agent that's otherwise very often used in abusive requests by malicious bots is leading to CC (understandably) being blocked from useful resources, which is quite a shame.

Indeed, I blocked WebFetch completely and set up this script which will use my real chrome with cookies and all, so it can look at websites I'm logged in on and it has a --no-headless option to actually show the chrome window in case I need to manually go through any captchas, etc. It also outputs websites in markdown so it's gentle on the context window and can list attachments like images / pdfs / videos / etc on the page. It also caches results as Claude tends to re-fetch the same URL multiple times:

#!/usr/bin/env node
/**
 * Browser automation wrapper for fetching content that bypasses bot protection
 * Uses Patchright (patched Playwright) to handle Cloudflare, DataDome and other challenges
 *
 * Usage:
 *   bin/fetch_claude <url>                    - Fetch full HTML
 *   bin/fetch_claude <url> --text             - Convert to Markdown (links, tables, code, etc.)
 *   bin/fetch_claude <url> --attachments      - Extract all attachment URLs in document order (images, gifs, videos, pdfs)
 *   bin/fetch_claude <url> --screenshot <path> - Save screenshot
 *   bin/fetch_claude <url> --download <path>  - Download binary file (images, PDFs, etc)
 *   bin/fetch_claude <url> --wait <selector>  - Wait for element before fetching
 *   bin/fetch_claude <url> --timeout <ms>     - Set timeout (default: 30000)
 *   bin/fetch_claude <url> --no-cache         - Skip cache, force fresh fetch
 *   bin/fetch_claude <url> --no-headless      - Show browser window (for debugging/manual CAPTCHA)
 *
 * Examples:
 *   bin/fetch_claude https://www.forbes.com/article --text
 *   bin/fetch_claude https://example.com --attachments  # Get all attachment URLs in order
 *   bin/fetch_claude https://example.com --screenshot ./page.png
 *   bin/fetch_claude https://example.com --download ./image.png
 *   bin/fetch_claude https://example.com --wait ".main-content"
 *   bin/fetch_claude https://example.com --no-headless  # Shows browser, waits 90s for login/CAPTCHA
 *
 * Browser profile:
 *   Cookies and session data persist in ~/.claude/tmp/browser_profile/
 *   Log in once with --no-headless, subsequent runs stay logged in
 *
 * Caching:
 *   Results are cached in ./tmp/fetch_cache/ for 2 hours
 *   Cache key includes URL and options (--text, --wait, --attachments)
 */

const { chromium } = require('patchright');
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const TurndownService = require('turndown');
const { gfm } = require('@joplin/turndown-plugin-gfm');

function getCacheKey(url, options) {
  const key = JSON.stringify({ url, ...options });
  return crypto.createHash('sha256').update(key).digest('hex');
}

function getCachePath(cacheKey) {
  const cacheDir = path.join(__dirname, '..', 'tmp', 'fetch_cache');
  if (!fs.existsSync(cacheDir)) {
    fs.mkdirSync(cacheDir, { recursive: true });
  }
  return path.join(cacheDir, `${cacheKey}.cache`);
}

function readCache(cacheKey, maxAge = 7200000) { // 2 hours default
  const cachePath = getCachePath(cacheKey);

  if (!fs.existsSync(cachePath)) {
    return null;
  }

  const stats = fs.statSync(cachePath);
  const age = Date.now() - stats.mtimeMs;

  if (age > maxAge) {
    fs.unlinkSync(cachePath);
    return null;
  }

  return fs.readFileSync(cachePath, 'utf8');
}

function writeCache(cacheKey, content) {
  const cachePath = getCachePath(cacheKey);
  fs.writeFileSync(cachePath, content, 'utf8');
}

async function main() {
  const args = process.argv.slice(2);

  if (args.length === 0 || args[0] === '--help' || args[0] === '-h') {
    console.error('Usage: bin/fetch_claude <url> [options]');
    console.error('Options:');
    console.error('  --text              Convert to Markdown (links, tables, code blocks, images)');
    console.error('  --attachments       Extract all attachment URLs in document order (images, gifs, videos, pdfs)');
    console.error('  --screenshot <path> Save screenshot to file');
    console.error('  --download <path>   Download binary file (images, PDFs, etc)');
    console.error('  --wait <selector>   Wait for CSS selector before fetching');
    console.error('  --timeout <ms>      Set timeout in milliseconds (default: 30000)');
    console.error('  --no-cache          Skip cache, force fresh fetch');
    console.error('  --no-headless       Show browser window (for debugging/manual CAPTCHA)');
    console.error('  --pause <seconds>   Manual interaction time in non-headless mode (default: 30)');
    process.exit(args.length === 0 ? 1 : 0);
  }

  const url = args[0];
  let textOnly = false;
  let attachmentsOnly = false;
  let screenshot = null;
  let download = null;
  let waitFor = null;
  let timeout = 30000;
  let useCache = true;
  let headless = true;
  let pauseSeconds = 30;

  // Parse options
  for (let i = 1; i < args.length; i++) {
    if (args[i] === '--text') {
      textOnly = true;
    } else if (args[i] === '--attachments') {
      attachmentsOnly = true;
    } else if (args[i] === '--screenshot' && args[i + 1]) {
      screenshot = args[++i];
    } else if (args[i] === '--download' && args[i + 1]) {
      download = args[++i];
    } else if (args[i] === '--wait' && args[i + 1]) {
      waitFor = args[++i];
    } else if (args[i] === '--timeout' && args[i + 1]) {
      timeout = parseInt(args[++i]);
    } else if (args[i] === '--no-cache') {
      useCache = false;
    } else if (args[i] === '--no-headless') {
      headless = false;
    } else if (args[i] === '--pause' && args[i + 1]) {
      pauseSeconds = parseInt(args[++i]);
    }
  }

  // Handle binary file downloads (simple HTTP, no browser needed)
  if (download) {
    try {
      const https = require('https');
      const http = require('http');
      const fileStream = fs.createWriteStream(download);
      const protocol = url.startsWith('https') ? https : http;

      protocol.get(url, (response) => {
        response.pipe(fileStream);
        fileStream.on('finish', () => {
          fileStream.close();
          console.error(`Downloaded to: ${download}`);
          process.exit(0);
        });
      }).on('error', (error) => {
        fs.unlink(download, () => {});
        console.error(`Error downloading ${url}:`, error.message);
        process.exit(1);
      });
      return;
    } catch (error) {
      console.error(`Error downloading ${url}:`, error.message);
      process.exit(1);
    }
  }

  // Check cache (skip if screenshot requested, cache disabled, or non-headless mode)
  if (useCache && !screenshot && headless) {
    const cacheKey = getCacheKey(url, { textOnly, attachmentsOnly, waitFor });
    const cached = readCache(cacheKey);

    if (cached) {
      console.log(cached);
      process.exit(0);
    }
  }


  // Use persistent browser context to preserve cookies/sessions between runs
  const profileDir = path.join(process.env.HOME, '.claude', 'tmp', 'browser_profile');
  if (!fs.existsSync(profileDir)) {
    fs.mkdirSync(profileDir, { recursive: true });
  }

  const context = await chromium.launchPersistentContext(profileDir, {
    headless: headless,
    channel: 'chrome',  // Use system Chrome for proper userAgentData branding
    viewport: { width: 1920, height: 1080 },
    args: [
      '--disable-blink-features=AutomationControlled',
      '--no-sandbox',
      '--window-size=1200,800',
    ],
  });

  try {
    const page = await context.newPage();
    page.setDefaultTimeout(timeout);

    // Navigate to the page
    await page.goto(url, {
      waitUntil: 'domcontentloaded',
      timeout: timeout
    });

    // Wait for specific selector if requested
    if (waitFor) {
      await page.waitForSelector(waitFor, { timeout: timeout });
    } else {
      // In non-headless mode, pause for manual interaction (login, CAPTCHA, etc.)
      if (!headless) {
        console.error(`Browser visible — you have ${pauseSeconds} seconds to log in or solve CAPTCHAs...`);
        console.error('The page will be captured after the wait period.');
        await page.waitForTimeout(pauseSeconds * 1000);
      }

      // Give page JS time to settle
      await page.waitForTimeout(2000);

      // Scroll to trigger lazy-loaded content
      await page.evaluate(async () => {
        const delay = ms => new Promise(r => setTimeout(r, ms));
        const scrollStep = Math.max(400, Math.floor(window.innerHeight * 0.8));
        let lastHeight = 0;
        for (let i = 0; i < 20; i++) {
          window.scrollBy(0, scrollStep);
          await delay(300);
          const currentHeight = document.documentElement.scrollTop;
          if (currentHeight === lastHeight) break;
          lastHeight = currentHeight;
        }
        window.scrollTo(0, 0);
      });
      await page.waitForTimeout(500);
    }

    // Take screenshot if requested
    if (screenshot) {
      await page.screenshot({ path: screenshot, fullPage: true });
      console.error(`Screenshot saved to: ${screenshot}`);
    }

    // Extract content
    let content;
    if (textOnly) {
      // Get cleaned HTML, removing noise elements
      const html = await page.evaluate(() => {
        // Always start with body and remove noise (exclusion-based approach)
        const clone = document.body.cloneNode(true);

        // Remove noise elements that don't contribute to main content
        const noiseSelectors = [
          // Technical elements
          'script', 'style', 'noscript', 'iframe', 'svg', 'template',
          // Structural noise
          'nav', 'footer', 'header', 'aside',
          '[role="navigation"]', '[role="banner"]', '[role="contentinfo"]', '[role="complementary"]',
          '.nav', '.navbar', '.footer', '.header', '.sidebar', '.menu',
          // Cookie/GDPR consent
          '.gdpr-wrapper', '.cookie-banner', '.cookie-consent', '.cookie-notice',
          '[class*="cookie"]', '[class*="consent"]', '[id*="cookie"]', '[id*="consent"]',
          '.onetrust-consent', '#onetrust-banner-sdk', '.cc-banner',
          // Ads and social
          '.advertisement', '.ad', '.ads', '.social-share', '.related-posts',
          '.share-buttons', '.social-buttons', '.author-bio', '.newsletter-signup',
          '.trending', '.popular-posts', '.recommended', '.more-stories',
          '[aria-hidden="true"]', '[data-ad]', '[data-advertisement]',
          '.breadcrumb', '.breadcrumbs', '.pagination', '.tags-list',
          // Popups and modals (but not product content)
          '.modal-backdrop', '.overlay:not(.product-overlay)',
          // Skip links and accessibility helpers
          '.skip-link', '.screen-reader-text', '.sr-only'
        ];
        noiseSelectors.forEach(selector => {
          clone.querySelectorAll(selector).forEach(el => el.remove());
        });

        // Remove elements that are purely decorative (icon-only links, etc)
        clone.querySelectorAll('a').forEach(a => {
          const text = a.textContent.trim();
          // Remove links with no text or only whitespace/special chars
          if (!text || /^[\s\u200B-\u200D\uFEFF]*$/.test(text)) {
            a.remove();
          }
        });

        return clone.innerHTML;
      });

      // Configure Turndown for clean Markdown output
      const turndown = new TurndownService({
        headingStyle: 'atx',           // # style headings
        hr: '---',
        bulletListMarker: '-',
        codeBlockStyle: 'fenced',      // ``` style code blocks
        fence: '```',
        emDelimiter: '*',
        strongDelimiter: '**',
        linkStyle: 'inlined',          // [text](url) style
        linkReferenceStyle: 'full'
      });

      // Add GFM support (tables, strikethrough, task lists)
      turndown.use(gfm);

      // Treat custom elements (web components) as block containers
      // Without this, Turndown treats unknown elements as inline and drops block children like <p>
      turndown.addRule('customElements', {
        filter: (node) => node.nodeName.includes('-'),
        replacement: (content) => content ? '\n\n' + content + '\n\n' : ''
      });

      // Remove empty links and clean up
      turndown.addRule('removeEmptyLinks', {
        filter: (node) => node.nodeName === 'A' && !node.textContent.trim(),
        replacement: () => ''
      });

      // Better image handling - include alt text and src
      turndown.addRule('images', {
        filter: 'img',
        replacement: (content, node) => {
          const alt = node.alt || '';
          const src = node.src || '';
          const title = node.title ? ` "${node.title}"` : '';
          if (!src) return '';
          return `![${alt}](${src}${title})`;
        }
      });

      content = turndown.turndown(html);

      // Clean up the markdown output
      content = content
        // Remove icon-only links like [**](url), [****](url), [**Tweet](url)
        .replace(/\[\*{1,4}[A-Za-z]*\]\([^)]+\)/g, '')
        // Remove standalone ** (empty bold from icons)
        .replace(/^\*{2,4}$/gm, '')
        // Remove lines starting with SHARE followed by links/icons
        .replace(/^SHARE\s*(\[\*{1,4}\][^\n]*)*$/gm, '')
        // Remove lines that are just social labels
        .replace(/^(Tweet|Pin|Email)\s*$/gim, '')
        // Truncate at common noise markers
        .replace(/\n(Found this article interesting\?|Trending News|Related Posts|Popular Posts|More Stories|You May Also Like|Recommended|Leave a Reply|Read More|More From Forbes|Also Read)[^\n]*[\s\S]*$/i, '')
        // Remove trailing related article links (### [Title](url) at end of article)
        .replace(/(\n### \[[^\]]+\]\([^)]+\)\s*)+$/g, '')
        // Max 3 consecutive newlines
        .replace(/\n{4,}/g, '\n\n\n')
        // Remove whitespace-only lines
        .replace(/^[\t ]+$/gm, '')
        .trim();
    } else if (attachmentsOnly) {
      // Extract all attachment URLs in document order
      const attachments = await page.evaluate(() => {
        const urls = [];
        const seen = new Set();

        // Common attachment extensions
        const attachmentExtensions = /\.(png|jpg|jpeg|gif|webp|svg|mp4|webm|mov|avi|pdf|doc|docx|xls|xlsx|ppt|pptx|zip|rar|7z)$/i;

        // Walk the DOM in document order to get attachments
        const walker = document.createTreeWalker(
          document.body,
          NodeFilter.SHOW_ELEMENT,
          null,
          false
        );

        let node;
        while (node = walker.nextNode()) {
          let url = null;

          // Images
          if (node.tagName === 'IMG' && node.src) {
            url = node.src;
          }
          // Video sources
          else if (node.tagName === 'VIDEO' && node.src) {
            url = node.src;
          }
          else if (node.tagName === 'SOURCE' && node.src) {
            url = node.src;
          }
          // Links to downloadable files
          else if (node.tagName === 'A' && node.href && attachmentExtensions.test(node.href)) {
            url = node.href;
          }
          // Embedded objects
          else if (node.tagName === 'EMBED' && node.src) {
            url = node.src;
          }
          else if (node.tagName === 'OBJECT' && node.data) {
            url = node.data;
          }

          // Add if valid and not a duplicate
          if (url && !seen.has(url)) {
            // Skip data: URLs, tiny tracking pixels, avatars, favicons, social icons
            if (url.startsWith('data:')) continue;
            if (url.includes('avatar')) continue;
            if (url.includes('favicon')) continue;
            if (url.includes('pixel')) continue;
            if (url.includes('tracking')) continue;
            if (url.includes('social-')) continue;
            if (url.includes('/icon:')) continue;

            seen.add(url);
            urls.push(url);
          }
        }

        return urls;
      });

      content = attachments.join('\n');
    } else {
      content = await page.content();
    }

    // Save to cache (if not screenshot-only request)
    if (useCache && !screenshot) {
      const cacheKey = getCacheKey(url, { textOnly, attachmentsOnly, waitFor });
      writeCache(cacheKey, content);
    }

    console.log(content);
    await context.close();
    process.exit(0);
  } catch (error) {
    console.error(`Error fetching ${url}:`, error.message);
    await context.close();
    process.exit(1);
  }
}

main();

Got blocked again two or three times today.

Last block was on this:

https://developer.apple.com/documentation/safari-release-notes/safari-26_3-release-notes

Adding a real-world case for this: we use Cloudflare Super Bot Fight Mode and added a
bypass rule matching User-Agent contains "Claude" — which worked for
Claude web chat. But Claude Code's WebFetch still gets blocked with a 403 because it's
sending axios/1.13.4 as the User-Agent, at least in the VS Code extension.

This means even sites that are actively trying to allowlist Claude can't do so for
Claude Code without also allowlisting the generic axios UA, which defeats the purpose
of bot protection.

A proper UA like ClaudeCode/1.x (Anthropic) would fix this and better align with how
ClaudeBot already identifies itself for web crawling.

In CC v2.1.83, it's sending Claude-User (claude-code/2.1.83; +https://support.anthropic.com/) as a UA.

@aveao confirmed UA is now sending Claude-User (claude-code/2.1.83; +https://support.anthropic.com/) after updating Claude Code extension to v2.1.83 - thank you!!

Same here; confirmed.

This issue was fixed as of version 2.1.83.

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.