agentspeed.
← all tools
Tool

robots.txt agent checker

Paste any URL. We fetch its /robots.txt, parse it, and tell you exactly which AI agents are allowed or blocked. Fifteen agents covered, including GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Bytespider, FacebookBot, Googlebot, and Bingbot.

Tool
›
stripe.comfetched https://stripe.com/robots.txtHTTP 20015 allowed0 blocked
agentverdictreason
GPTBotallowedno matching rule (default allow)
ChatGPT-Userallowedno matching rule (default allow)
ClaudeBotallowedno matching rule (default allow)
PerplexityBotallowedno matching rule (default allow)
Google-Extendedallowedno matching rule (default allow)
Applebot-Extendedallowedno matching rule (default allow)
CCBotallowedno matching rule (default allow)
Amazonbotallowedno matching rule (default allow)
Bytespiderallowedno matching rule (default allow)
FacebookBotallowedno matching rule (default allow)
Googlebotallowedno matching rule (default allow)
Bingbotallowedno matching rule (default allow)
cohere-aiallowedno matching rule (default allow)
Meta-ExternalAgentallowedno matching rule (default allow)
OAI-SearchBotallowedno matching rule (default allow)

Sitemaps declared

  • https://stripe.com/sitemap/sitemap.xml

Raw robots.txt

Sitemap: https://stripe.com/sitemap/sitemap.xml

User-agent: ia_archiver
Allow: /docs/api
Allow: /docs/api$
Disallow: /docs
Disallow: /docs$
Disallow: /bitcoin/refund
Disallow: /sources/refund
Disallow: /sources/sepa_mandate
Disallow: /sources/test_source
Disallow: /sources/test_klarna
Disallow: /handoff-healthcheck
Disallow: /handoff

User-agent: *
Allow: /docs
Allow: /docs$
Disallow: /bitcoin/refund
Disallow: /sources/refund
Disallow: /sources/sepa_mandate
Disallow: /sources/test_source
Disallow: /sources/test_klarna
Disallow: /unsupported-browser
Disallow: /handoff-healthcheck
Disallow: /handoff

User-agent: rogerbot
Crawl-delay: 2
Watch this URL — get alerted when this changes

Turn this one-off check into a scheduled watch. We re-run it, diff the result, and alert you on a regression through the same channels as your agent alerts. Watches are a paid feature.

Manage watches at /watches. You’ll be asked to sign in if you haven’t already.

Examples

Click an example to run the tool with pre-filled inputs.

Why this matters for AI agents

Most sites inherit a wildcard User-agent: * rule into AI traffic without realising. ChatGPT, Claude, and Perplexity respect explicit agent rules first; if you have blocked *, you have blocked them too.

Agents that get a 403 or a wildcard disallow do not appear in their answers. AgentSpeed scores every site against robots_txt.allows_known_agents. Passing it is the cheapest single change you can make for AI visibility.

Common fixes
Wildcard disallow blocks every AI crawler
Replace User-agent: *\nDisallow: / with named entries for each agent you actually want to block. Or keep the wildcard and add agent-specific Allow: / blocks below it.
Want search but not training
Add Content-Signal directives: Allow-Crawl-For-Search, Disallow-AI-Training. Compliant agents will honour the policy.
Block Bytespider but keep Google
Add an explicit User-agent: Bytespider\nDisallow: / block. Wildcard rules then no longer apply to Bytespider.
/llms.txt accidentally blocked
Make sure no Disallow: /llms.txt rule exists. Agents that follow robots.txt strictly will skip the file if it is disallowed.
Linked rubric checks
FAQ
Why do I need to allow GPTBot?
GPTBot is OpenAI’s crawler used to discover and ground answers from your site. Blocking it means ChatGPT will not cite you when users ask about your product.
Does blocking Bytespider affect Google?
No. Bytespider is ByteDance’s crawler. It has nothing to do with Googlebot. You can block one without affecting the other. That is exactly what agent-specific rules are for.
What is Content-Signal?
Content-Signal is a Cloudflare-led extension to robots.txt that lets you grant fine-grained consent: search vs inference vs training vs summarisation. Compliant agents honour the directives. We surface them when present.
Will allowing AI bots affect my SEO?
No. Allowing GPTBot, ClaudeBot, etc. is independent of how Googlebot indexes your site. Most SEO impact comes from how your content is structured, not which crawlers you allow.
Why does my site allow GPTBot but not ClaudeBot?
You have an agent-specific Disallow for ClaudeBot somewhere, usually inherited from a copy-pasted snippet. Audit your robots.txt for any User-agent: ClaudeBot blocks.
Can I see the robots.txt other sites publish?
Yes. /robots.txt is public by definition. Try the example URLs above to compare how OpenAI, GitHub, and Stripe configure theirs.
Run a full agent-readiness scan
Full scan

Score the entire site against 30 agent-readiness checks across discoverability, readability, structured data, actionability, and performance.

Get the full report for stripe.com →