← all toolsTool
robots.txt agent checker
Paste any URL. We fetch its /robots.txt, parse it, and tell you exactly which AI agents are allowed or blocked. Fifteen agents covered, including GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Bytespider, FacebookBot, Googlebot, and Bingbot.
Tool
Examples
Click an example to run the tool with pre-filled inputs.
Why this matters for AI agents
Most sites inherit a wildcard User-agent: * rule into AI traffic without realising. ChatGPT, Claude, and Perplexity respect explicit agent rules first; if you have blocked *, you have blocked them too.
Agents that get a 403 or a wildcard disallow do not appear in their answers. AgentSpeed scores every site against robots_txt.allows_known_agents. Passing it is the cheapest single change you can make for AI visibility.
Common fixes
Wildcard disallow blocks every AI crawlerReplace User-agent: *\nDisallow: / with named entries for each agent you actually want to block. Or keep the wildcard and add agent-specific Allow: / blocks below it.
Want search but not trainingAdd Content-Signal directives: Allow-Crawl-For-Search, Disallow-AI-Training. Compliant agents will honour the policy.
Block Bytespider but keep GoogleAdd an explicit User-agent: Bytespider\nDisallow: / block. Wildcard rules then no longer apply to Bytespider.
/llms.txt accidentally blockedMake sure no Disallow: /llms.txt rule exists. Agents that follow robots.txt strictly will skip the file if it is disallowed.
Linked rubric checks
- robots_txt.allows_known_agents
reachability.statusThe site responds 200 to AgentSpeedBot. Robots.txt blocks would surface here too.
reachability.status
FAQ
Why do I need to allow GPTBot?
GPTBot is OpenAI’s crawler used to discover and ground answers from your site. Blocking it means ChatGPT will not cite you when users ask about your product.
Does blocking Bytespider affect Google?
No. Bytespider is ByteDance’s crawler. It has nothing to do with Googlebot. You can block one without affecting the other. That is exactly what agent-specific rules are for.
What is Content-Signal?
Content-Signal is a Cloudflare-led extension to robots.txt that lets you grant fine-grained consent: search vs inference vs training vs summarisation. Compliant agents honour the directives. We surface them when present.
Will allowing AI bots affect my SEO?
No. Allowing GPTBot, ClaudeBot, etc. is independent of how Googlebot indexes your site. Most SEO impact comes from how your content is structured, not which crawlers you allow.
Why does my site allow GPTBot but not ClaudeBot?
You have an agent-specific Disallow for ClaudeBot somewhere, usually inherited from a copy-pasted snippet. Audit your robots.txt for any User-agent: ClaudeBot blocks.
Can I see the robots.txt other sites publish?
Yes. /robots.txt is public by definition. Try the example URLs above to compare how OpenAI, GitHub, and Stripe configure theirs.
Run a full agent-readiness scan
Full scanScore the entire site against 30 agent-readiness checks across discoverability, readability, structured data, actionability, and performance.
Get the full report →