ClaudeBot is blocked by your robots.txt

Anthropic's crawler is disallowed, so Claude cannot read your pages or cite them in answers.

ClaudeBot is the user-agent Anthropic uses to fetch pages on behalf of Claude (the Anthropic API and the Claude.ai chat product). When robots.txt contains "User-agent: ClaudeBot" followed by "Disallow: /", every Claude request for your content gets refused at the robots.txt layer, and no HTML is ever fetched.

Claude is one of the most-used assistants for technical, research, and long-context work, exactly the kinds of queries where citing your docs, your blog, or your product pages compounds in value. Anthropic respects robots.txt strictly, so a single Disallow line takes you out of the answer set. Unlike search engines, there is no fallback: when ClaudeBot can't fetch, Claude doesn't show your content at all. There is no "submit URL" path that bypasses robots.

If a developer asks Claude 'what's the best Postgres-on-Kubernetes operator' and you've blocked ClaudeBot, your product is not in the comparison. Your competitor is. Unblocking is a one-line change and has no SEO impact (Google uses Googlebot, not ClaudeBot). Sites that unblock typically see Anthropic-referred traffic appear in their analytics within 1–2 weeks as the index refreshes.

Open https://your-domain.com/robots.txt and search for "ClaudeBot" (case-sensitive). If you see "User-agent: ClaudeBot" followed by "Disallow: /" (or a global "User-agent: *" with "Disallow: /" and no ClaudeBot allow rule above it), Claude is blocked. Note: there are two Anthropic user-agents to consider, ClaudeBot (training and indexing) and Claude-User (on-demand fetches when a Claude user pastes a URL). Most sites want to allow both.

Remove any "User-agent: ClaudeBot" + "Disallow: /" stanza. If you have a wildcard disallow you want to keep, add a specific "User-agent: ClaudeBot" + "Allow: /" block above the wildcard so the more-specific rule wins. Add the same block for Claude-User. While you are editing, also confirm GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, and Google-Extended are allowed, since the same robots.txt failure pattern affects all of them.

# Allow Anthropic crawlers and other major AI agents.
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: Claude-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Default for everyone else.
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://your-domain.com/sitemap.xml

The reference snippet above is platform-agnostic. Below are the concrete steps for the four most common stacks plus a generic fallback for any other host.

import type { MetadataRoute } from 'next';

const AI_AGENTS = [
  'ClaudeBot',
  'Claude-User',
  'GPTBot',
  'OAI-SearchBot',
  'ChatGPT-User',
  'PerplexityBot',
  'Google-Extended',
] as const;

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      ...AI_AGENTS.map((userAgent) => ({
        userAgent,
        allow: '/',
        disallow: ['/admin/', '/api/'],
      })),
      { userAgent: '*', allow: '/', disallow: ['/admin/'] },
    ],
    sitemap: 'https://your-domain.com/sitemap.xml',
  };
}

add_filter( 'robots_txt', function ( $output ) {
  $agents = [
    'ClaudeBot', 'Claude-User',
    'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
    'PerplexityBot', 'Google-Extended',
  ];
  $rules = '';
  foreach ( $agents as $agent ) {
    $rules .= "User-agent: {$agent}\nAllow: /\nDisallow: /wp-admin/\n\n";
  }
  return $rules . $output;
}, 10, 1 );

# AI agents
{%- assign ai_agents = "ClaudeBot,Claude-User,GPTBot,OAI-SearchBot,ChatGPT-User,PerplexityBot,Google-Extended" | split: "," -%}
{%- for agent in ai_agents -%}
User-agent: {{ agent }}
Allow: /
Disallow: /admin/

{% endfor -%}

{%- for group in robots.default_groups -%}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{%- if group.sitemap -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap.xml

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap.xml

• Allowing only ClaudeBot but not Claude-User. ClaudeBot indexes the web for Claude.ai's knowledge; Claude-User fetches when a Claude user explicitly references a URL. Most sites want both.
• Adding the Allow rule below a wildcard Disallow. Strict robots.txt parsers use longest-match, so keep specific rules above the wildcard to avoid edge-case parsing differences.
• Blocking ClaudeBot for the same misconception as GPTBot, "to save bandwidth." Anthropic crawlers are well-behaved and respect crawl-delay; the citations are typically worth the fetch cost many times over.
• Confusing Claude-Web (an old, deprecated user-agent) with the current names. The active user-agents from Anthropic are ClaudeBot and Claude-User. Update any old robots.txt rules referring to Claude-Web; they are no-ops now.
• Adding a noindex meta tag and assuming it blocks ClaudeBot. It does not; robots.txt is the only mechanism Anthropic checks before fetching.

After publishing, fetch https://your-domain.com/robots.txt and confirm ClaudeBot has an Allow rule. Then run a fresh AgentSpeed scan, and the "robots.txt allows known agents" check should pass within seconds. Anthropic-referred traffic begins appearing in analytics 1–2 weeks later as the index refreshes.