agentspeed.
← all fixes
Remediation

robots.txt does not disallow well-known agent user-agents (GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended).

robots_txt.allows_known_agentsdiscoverabilityweight 2deterministicauto-fix available
Why it matters

Most sites inherit a wildcard User-agent: * Disallow rule into AI traffic without realising. ChatGPT, Claude, and Perplexity respect explicit agent rules first; if you have blocked *, you have blocked them too. The cheapest single change for AI visibility is allowing the named AI agents you want crawling your site.

Underlying specification: https://datatracker.ietf.org/doc/html/rfc9309

How to fix
  1. Run the robots.txt agent checker tool to see exactly which AI agents your live robots.txt allows or blocks.
  2. For each AI agent currently blocked, add an explicit `User-agent: <Bot>` block followed by `Allow: /` at the bottom of robots.txt.
  3. Place agent-specific blocks AFTER the wildcard rule so they take precedence.
  4. Validate again. The checker emits a paste-ready snippet for the blocked agents.
Generated snippet

The snippet below uses your-domain.example as a placeholder. Replace it with your actual host before deploying. For a fix tailored to your live site, run a scan and open the per-domain remediation page.

Generated fixreplace file→ https://your-domain.example/robots.txt

Your current /robots.txt disallows these AI agents: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended. Replace your robots.txt with the version below to explicitly allow them while keeping any custom rules you've added below the wildcard block.

# robots.txt for your-domain.example
# Explicitly allow well-known AI agents that were previously blocked.
# Adjust or remove specific agents as your policy requires.

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

Sitemap: https://your-domain.example/sitemap.xml
save as https://your-domain.example/robots.txt
  • ›This snippet replaces your existing robots.txt. Merge any custom Disallow rules you have today below the wildcard block.
  • ›Cloudflare and a few CDNs cache robots.txt aggressively, so purge the cache after deploy to confirm AI agents see the change.
FAQ
Why do I need to allow GPTBot?

GPTBot is OpenAI’s crawler used to discover and ground answers from your site. Blocking it means ChatGPT will not cite your content when users ask about your product.

Does blocking Bytespider affect Google?

No. Bytespider is ByteDance’s crawler. It has nothing to do with Googlebot. You can block one without affecting the other. That is what agent-specific rules are for.

Will allowing AI bots affect my SEO?

No. Allowing GPTBot, ClaudeBot, etc. is independent of how Googlebot indexes your site. AI bots and search bots are governed by separate User-agent rules.

Run a scan to check your site
Verify on your site

Run a scan against your domain to see whether robots_txt.allows_known_agents currently passes, warns, or fails. Free, no signup; the score page is permanent and citable.

Run a scan →See the full rubric →