← all toolsTool
Are you blocking AI agents?
Check whether your robots.txt allows or blocks ChatGPT, Claude, Perplexity, Google AI, Apple, Amazon, and other AI crawlers. Ten agents covered, three paste-ready fix modes.
Tool
Examples
Click an example to run the tool with pre-filled inputs.
Why this matters for AI agents
Most AI crawlers respect robots.txt strictly. A single Disallow: / under the wrong user-agent silently removes you from every answer that crawler powers, and there is no fallback path back into the index.
The right policy is rarely “allow everything” or “block everything.” Decide which agents drive customer discovery (worth allowing) vs. which create training / IP risk with no citation upside (worth blocking). The three fix modes below cover the common positions.
Common fixes
Wildcard disallow blocks every agentReplace User-agent: *\nDisallow: / with named entries for each agent you actually want to block, or keep the wildcard and add agent-specific Allow: / blocks above it.
Allow citations, block trainingUse Balanced mode below: allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended; block CCBot, Bytespider, Applebot-Extended.
Block one crawler onlyAdd an explicit User-agent: Bytespider\nDisallow: / stanza above any wildcard. Specific user-agent rules win over *.
GPTBot and Google-Extended are independentAllowing Google-Extended does not affect search ranking. It is the AI-only Google crawler. Likewise blocking GPTBot does not affect Googlebot. Treat them separately.
Linked rubric checks
- robots_txt.allows_known_agents
FAQ
Should I just allow every AI crawler?
Not necessarily. Allow the agents that drive citations (ChatGPT, Claude, Perplexity, Google AI). Decide deliberately about training-only crawlers (CCBot, Applebot-Extended, Bytespider): they take your content into datasets without sending traffic back. The three fix modes below codify these positions.
Will allowing AI bots affect my SEO?
No. SEO is governed by Googlebot. The AI-specific user-agents (GPTBot, ClaudeBot, Google-Extended) are independent crawlers; allowing or blocking them is orthogonal to search ranking.
What is the difference between GPTBot and ChatGPT-User?
GPTBot indexes the web for OpenAI training and search. ChatGPT-User runs only when a ChatGPT user explicitly references a URL. Most sites want both allowed: GPTBot for citations in default answers, ChatGPT-User for on-demand fetches.
Why is Bytespider treated differently?
Bytespider has a reputation for ignoring crawl-delay and aggressive fetch behaviour, with no public citation surface for most non-Chinese sites. Many operators block it as the safe default; allow only if you actively target the TikTok / Doubao audience.
How often should I re-check?
Whenever you redeploy your site or update robots.txt. We recommend running the full AgentSpeed scan monthly so the result is timestamped and historically tracked.
Run a full agent-readiness scan
Full scanScore the entire site against 30 agent-readiness checks across discoverability, readability, structured data, actionability, and performance.
Get the full report →