GPTBot is blocked by your robots.txt

OpenAI's crawler is disallowed, so ChatGPT cannot read your pages or cite them in answers.

GPTBot is the user-agent OpenAI uses to fetch pages on behalf of ChatGPT and the OpenAI training corpus. When robots.txt contains "User-agent: GPTBot" followed by "Disallow: /", every ChatGPT request for your content gets refused at the robots.txt layer, before any HTML is fetched.

ChatGPT has hundreds of millions of weekly users and is one of the largest sources of agent-driven traffic on the internet. Blocking GPTBot means ChatGPT cannot cite your pages, cannot show your products in answers, and cannot recommend your docs. There is no second chance; the agent simply leaves your domain off the answer.

If a user asks ChatGPT 'what's the best CRM for small teams' and you've blocked GPTBot, you are not in the answer. Your competitor is. Unblocking is a one-line robots.txt change with no risk to SEO (Google uses Googlebot, not GPTBot). Sites that unblock typically see ChatGPT-referred traffic appear in analytics within 1–2 weeks as the crawler refreshes its index.

Open https://your-domain.com/robots.txt. Search for "GPTBot" (case-sensitive). If you see "User-agent: GPTBot" followed by "Disallow: /" (or a global "User-agent: *" with "Disallow: /" and no GPTBot allow rule), you are blocking ChatGPT. The fix is to remove the disallow or add an explicit allow.

Remove any "User-agent: GPTBot" + "Disallow: /" block from robots.txt. If you have a global wildcard disallow you want to keep, add an explicit "User-agent: GPTBot" / "Allow: /" stanza above it so the more-specific rule wins. The same pattern applies to OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended.

# Allow major AI agents to read public pages.
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Default for everyone else.
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://your-domain.com/sitemap.xml

The reference snippet above is platform-agnostic. Below are the concrete steps for the four most common stacks plus a generic fallback for any other host.

import type { MetadataRoute } from 'next';

const AI_AGENTS = [
  'GPTBot',
  'OAI-SearchBot',
  'ChatGPT-User',
  'ClaudeBot',
  'PerplexityBot',
  'Google-Extended',
] as const;

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      ...AI_AGENTS.map((userAgent) => ({
        userAgent,
        allow: '/',
        disallow: ['/admin/', '/api/'],
      })),
      { userAgent: '*', allow: '/', disallow: ['/admin/'] },
    ],
    sitemap: 'https://your-domain.com/sitemap.xml',
  };
}

add_filter( 'robots_txt', function ( $output ) {
  $agents = [ 'GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'ClaudeBot', 'PerplexityBot', 'Google-Extended' ];
  $rules = '';
  foreach ( $agents as $agent ) {
    $rules .= "User-agent: {$agent}\nAllow: /\nDisallow: /wp-admin/\n\n";
  }
  return $rules . $output;
}, 10, 1 );

# AI agents
{%- assign ai_agents = "GPTBot,OAI-SearchBot,ChatGPT-User,ClaudeBot,PerplexityBot,Google-Extended" | split: "," -%}
{%- for agent in ai_agents -%}
User-agent: {{ agent }}
Allow: /
Disallow: /admin/

{% endfor -%}

{%- for group in robots.default_groups -%}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{%- if group.sitemap -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap.xml

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap.xml

• Using "Disallow:" with no path. That means "block nothing," which is the opposite of what most people intend. To block, use "Disallow: /". To allow everything, use "Allow: /" or omit the disallow entirely.
• Putting the AI-agent allow stanza below a global wildcard disallow. robots.txt resolves the most-specific user-agent block, but some legacy parsers read top-down, so keep specific rules above the wildcard to be safe.
• Blocking GPTBot to "save bandwidth." GPTBot honours crawl-delay and is well-behaved; the citations it generates typically more than pay for the fetch cost.
• Forgetting OAI-SearchBot and ChatGPT-User. GPTBot indexes for training; the other two fetch on-demand when a user asks ChatGPT a question. You usually want all three allowed.
• Adding a noindex meta tag and assuming it blocks GPTBot. It does not; robots.txt is the only mechanism agents check before fetching.

After publishing, fetch https://your-domain.com/robots.txt and confirm GPTBot has an Allow rule. Then run a fresh AgentSpeed scan, and the "robots.txt allows known agents" check should pass within seconds. ChatGPT-referred traffic begins appearing in analytics 1–2 weeks later as the index refreshes.