Check whether your robots.txt allows or blocks ChatGPT, Claude, Perplexity, Google AI, Apple, Amazon, and other AI crawlers. Ten agents covered, three paste-ready fix modes.
OpenAI's training and indexing crawler. Fetches public web content used to train and refresh ChatGPT's general knowledge.
Recommendation: Allow, Required for ChatGPT to cite your pages in answers. Block only if you have a strict no-training policy and accept lower citation visibility.
Training crawlerAllowed
ChatGPT-UserOpenAI
On-demand fetcher: runs when a ChatGPT user explicitly references a URL or asks ChatGPT to browse. Does not train models.
Recommendation: Allow, High-value, low-volume traffic. Blocks here mean ChatGPT can not retrieve your page even when a user asks for it by name.
On-demand user agentAllowed
ClaudeBotAnthropic
Anthropic's primary crawler. Indexes content used by Claude.ai and the Anthropic API for both training and live retrieval.
Recommendation: Allow, Required for Claude to cite your pages. Anthropic respects robots.txt strictly, so a single Disallow takes you out of every Claude answer.
Training + searchAllowed
Claude-UserAnthropic
On-demand fetcher: runs when a Claude user references a URL in a conversation. Independent of ClaudeBot.
Recommendation: Allow, Lets Claude follow user-provided URLs to your site. Low traffic, high intent, typically worth allowing.
On-demand user agentAllowed
PerplexityBotPerplexity
Perplexity's answer-engine crawler. Indexes pages so they can appear as cited sources in Perplexity answers.
Recommendation: Allow, Perplexity is one of the largest agent-driven referrers. Citations link directly to your URL, so blocking is rarely worth it.
Search / answer engineAllowed
Google-ExtendedGoogle
Google's separate user-agent for AI surfaces (Bard / Gemini / AI Overviews). Independent of Googlebot, so blocking here does NOT affect search ranking.
Recommendation: Allow, Required for inclusion in Google AI Overviews and Gemini answers. Independent of SEO; allowing has no downside on search rankings.
Training + searchAllowed
Applebot-ExtendedApple
Apple's AI training crawler. Distinct from Applebot (which powers Siri and Spotlight). Used for Apple Intelligence model training.
Recommendation: Consider blocking, Training-only signal with no direct citation surface today. Reasonable to allow if you favour wide AI inclusion; reasonable to block if you have IP concerns.
Training crawlerAllowed
CCBotCommon Crawl
Common Crawl is the upstream dataset used to train many LLMs (including some GPT and Llama generations). Crawls broadly, archives publicly.
Recommendation: Consider blocking, Bulk archival with no per-page citation back to your site. Block if you don't want your content in third-party training datasets.
Training crawlerAllowed
AmazonbotAmazon
Amazon's general-purpose crawler. Powers Alexa answers and feeds product/data pipelines across Amazon's services.
Recommendation: Allow, Commerce-relevant: Alexa and Amazon's agent surfaces benefit from access. Block if you sell on a competing platform and want to limit data flow.
Commerce agentAllowed
BytespiderByteDance
ByteDance's training crawler (TikTok / Doubao). Aggressive crawl rate; some sites have reported it ignoring crawl-delay directives.
Recommendation: Block unless needed, No public citation surface for most sites; aggressive fetch behaviour. Allow only if you actively target a Chinese-language audience.
Training crawlerAllowed
Paste-ready robots.txt
Three modes covering the common positions. Each is a complete robots.txt body. Replace the existing file at the root of your site, then re-check above.
Balanced modebalancedrobots.txt
Allow citation-driving agents (ChatGPT, Claude, Perplexity, Google AI). Block training-only crawlers with no direct visibility upside.
Allow every major AI crawler. Maximises citation surface across all answer engines and commerce agents. Right for sites that monetise through discovery.
# Discovery mode: maximise citation surface across major AI agents.
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: CCBot
Allow: /
User-agent: Amazonbot
Allow: /
# Bytespider is omitted by default. Its aggressive crawl rate often
# outweighs the discovery upside. Add it back if you target Chinese-
# language audiences via TikTok's Doubao.
User-agent: *
Allow: /
Sitemap: https://your-domain.com/sitemap.xml
Next steps
robots.txt is one signal of agent readiness. Run the full scan to see how the site scores across the full rubric, compare against a competitor, or set up monitoring to catch policy regressions.
Turn this one-off check into a scheduled watch. We re-run it, diff the result, and alert you on a regression through the same channels as your agent alerts. Watches are a paid feature.
Manage watches at /watches. You’ll be asked to sign in if you haven’t already.
Click an example to run the tool with pre-filled inputs.
Why this matters for AI agents
Most AI crawlers respect robots.txt strictly. A single Disallow: / under the wrong user-agent silently removes you from every answer that crawler powers, and there is no fallback path back into the index.
The right policy is rarely “allow everything” or “block everything.” Decide which agents drive customer discovery (worth allowing) vs. which create training / IP risk with no citation upside (worth blocking). The three fix modes below cover the common positions.
Common fixes
Wildcard disallow blocks every agent
Replace User-agent: *\nDisallow: / with named entries for each agent you actually want to block, or keep the wildcard and add agent-specific Allow: / blocks above it.
Add an explicit User-agent: Bytespider\nDisallow: / stanza above any wildcard. Specific user-agent rules win over *.
GPTBot and Google-Extended are independent
Allowing Google-Extended does not affect search ranking. It is the AI-only Google crawler. Likewise blocking GPTBot does not affect Googlebot. Treat them separately.
Each AI agent is explicitly allowed or, at minimum, not disallowed by wildcard.
robots_txt.allows_known_agents
FAQ
Should I just allow every AI crawler?
Not necessarily. Allow the agents that drive citations (ChatGPT, Claude, Perplexity, Google AI). Decide deliberately about training-only crawlers (CCBot, Applebot-Extended, Bytespider): they take your content into datasets without sending traffic back. The three fix modes below codify these positions.
Will allowing AI bots affect my SEO?
No. SEO is governed by Googlebot. The AI-specific user-agents (GPTBot, ClaudeBot, Google-Extended) are independent crawlers; allowing or blocking them is orthogonal to search ranking.
What is the difference between GPTBot and ChatGPT-User?
GPTBot indexes the web for OpenAI training and search. ChatGPT-User runs only when a ChatGPT user explicitly references a URL. Most sites want both allowed: GPTBot for citations in default answers, ChatGPT-User for on-demand fetches.
Why is Bytespider treated differently?
Bytespider has a reputation for ignoring crawl-delay and aggressive fetch behaviour, with no public citation surface for most non-Chinese sites. Many operators block it as the safe default; allow only if you actively target the TikTok / Doubao audience.
How often should I re-check?
Whenever you redeploy your site or update robots.txt. We recommend running the full AgentSpeed scan monthly so the result is timestamped and historically tracked.
Run a full agent-readiness scan
Full scan
Score the entire site against 30 agent-readiness checks across discoverability, readability, structured data, actionability, and performance.