The registry

24 agents tracked. 9 can move your score.

Every scan checks your robots.txt against this list, agent by agent. The tiers below are not a presentation choice — they are how the score is computed, and the other 15 agents are reported to you and deliberately not charged against your grade.

Tier 1 · 9 scored

Block one of these and you are out of AI answers

Answer-time fetchers and AI-search indexers with worldwide reach. Your robots_txt.allows_known_agents score is the share of these you allow, and nothing else feeds it.

Agent	Operator	Intent	robots.txt token
`ChatGPT-User`	OpenAI	Answering a user	`ChatGPT-User`
`OAI-SearchBot`	OpenAI	AI search index	`OAI-SearchBot`
`Claude-User`	Anthropic	Answering a user	`Claude-User`
`Claude-SearchBot`	Anthropic	AI search index	`Claude-SearchBot`
`Perplexity-User`	Perplexity	Answering a user	`Perplexity-User`
`PerplexityBot`	Perplexity	AI search index	`PerplexityBot`
`YouBot`	You.com	AI search index	`YouBot`
`Amazonbot`	Amazon	Answering a user	`Amazonbot`
`Applebot`	Apple	Answering a user	`Applebot`

Tier 2 · 10 reported, not scored

Blocking a training crawler is a licensing decision

These collect text to train models. Whether you allow them is a business call about your content, not a readiness defect — blocking every one of them does not change how well an agent can use your site, so it does not move your grade. The scan tells you which are blocked and stops there.

Agent	Operator	Intent	robots.txt token
`GPTBot`	OpenAI	Training data	`GPTBot`
`ClaudeBot`	Anthropic	Training data	`ClaudeBotanthropic-ai`
`Google-Extended`	Google	Training data	`Google-Extended`
`Applebot-Extended`	Apple	Training data	`Applebot-Extended`
`Bytespider`	ByteDance	Training data	`Bytespider`
`CCBot`	Common Crawl	Training data	`CCBot`
`cohere-ai`	Cohere	Training data	`cohere-ai`
`Diffbot`	Diffbot	Training data	`Diffbot`
`Meta-ExternalAgent`	Meta	Training data	`Meta-ExternalAgent`
`FacebookBot`	Meta	Training data	`FacebookBot`

Tier 3 · 5 checked conditionally

Regional engines, checked only if you say you want that market

Blocking Baidu is a defect for a shop selling into China and an entirely reasonable choice for a plumber in Ohio, and robots.txt alone cannot tell the two apart. So the scan reads what your own page declares — <html lang> and your hreflang alternates — and only reports these when your site says it wants those readers. Never scored: widening the denominator would penalise every site that never wanted those markets.

Agent	Operator	Intent	robots.txt token
`Baiduspider`	Baidu · China	AI search index	`Baiduspider`
`Sogou`	Sogou · China	AI search index	`Sogou web spiderSogou inst spiderSogou spider2`
`PetalBot`	Huawei · China	AI search index	`PetalBot`
`YandexBot`	Yandex · Russia	AI search index	`YandexBot`
`Yeti`	Naver · South Korea	AI search index	`Yeti`

Which of these can reach your site?

Free, 30 seconds, no signup — verdict per agent, plus the fix for each block.

Scan my site →

Every weight and threshold behind that verdict is published at /rubric, and our own measured error rate at /rubric/accuracy.