Tool

AI Crawler Access Matrix

Fetch any URL in parallel as ten AI crawlers and a default browser. See exactly which agents your site allows, blocks, redirects, challenges, or serves a different body to. Real HTTP fetches, real verdicts.

Tool

https://linode.com10 crawlers tested9 different1 blocked

agentverdictstatusbyteslatencysimilaritynote

GPTBotallowed · diff200161,6675325ms0.00OpenAI training crawler

ChatGPT-Userallowed · diff200161,6672209ms0.00ChatGPT browsing for a user

ClaudeBotallowed · diff200161,6672154ms0.00Anthropic crawler

Claude-Userallowed · diff200161,6673202ms0.00Claude browsing for a user

PerplexityBotallowed · diff200161,6675129ms0.00Perplexity citation crawler

Google-Extendedallowed · diff200161,6695238ms0.00Google AI training opt-out crawler

Googlebotallowed · diff200161,6674213ms0.00Google search index crawler

Bingbotallowed · diff200161,6674230ms0.00Bing search index crawler

CCBotallowed · diff200161,667392ms0.00Common Crawl, feeds many AI training datasets

Browserblocked4032,98280ms1.00Vanilla desktop Chrome, control row

What each row means

GPTBotallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
ChatGPT-Userallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
ClaudeBotallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
Claude-Userallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
PerplexityBotallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
Google-Extendedallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
Googlebotallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
Bingbotallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
CCBotallowed · diffAllowed (2xx) but the body differs substantially from the browser baseline. Possible cloaking, agent-aware rendering, or paywall variant.
BrowserblockedHTTP 403. The host is refusing this user-agent.

Result summary

Copyable summary for sharing with infra owners or filing an issue with your CDN.

AgentSpeed AI Crawler Access Matrix
Target: https://linode.com
Run: 2026-07-26T04:35:43.284Z

Per-agent verdict:
  - GPTBot           allowed · diff status=200 sim=0.00 bytes=161667
  - ChatGPT-User     allowed · diff status=200 sim=0.00 bytes=161667
  - ClaudeBot        allowed · diff status=200 sim=0.00 bytes=161667
  - Claude-User      allowed · diff status=200 sim=0.00 bytes=161667
  - PerplexityBot    allowed · diff status=200 sim=0.00 bytes=161667
  - Google-Extended  allowed · diff status=200 sim=0.00 bytes=161669
  - Googlebot        allowed · diff status=200 sim=0.00 bytes=161667
  - Bingbot          allowed · diff status=200 sim=0.00 bytes=161667
  - CCBot            allowed · diff status=200 sim=0.00 bytes=161667
  - Browser          blocked        status=403 sim=1.00 bytes=2982

Run again at: https://agentspeed.com/tools/ai-crawler-access-matrix

Watch this URL — get alerted when this changes

Turn this one-off check into a scheduled watch. We re-run it, diff the result, and alert you on a regression through the same channels as your agent alerts. Watches are a paid feature.

Manage watches at /watches. You’ll be asked to sign in if you haven’t already.

Examples

Click an example to run the tool with pre-filled inputs.

Why this matters for AI agents

Most agent-readiness problems show up here first. A site that blocks GPTBot at the WAF, redirects ChatGPT-User to /login, or serves a different body to PerplexityBot than to a browser will never show up in agent answers, and the only way to find out is to actually fetch as those agents.

This is the live equivalent of the journey.primary_probe check on a public scan, run on demand against any URL.

Common fixes

WAF blocks GPTBot

In Cloudflare bot management, allow GPTBot, ClaudeBot, and PerplexityBot explicitly. Their UA strings are documented and IPs are fixed.

Different body for bots than browser

If similarity drops below 0.80 between an agent and the Browser baseline, your CMS or edge layer is likely rendering a different variant. Audit any UA-conditional rendering rules, since agents often get the worst one.

Redirected to /login

Render server-side preview content for unauthenticated requests on landing pages. Agents cannot authenticate; they cite the auth wall.

Cloudflare challenge fires

Add the bot to Cloudflare’s allowed-bots list, or whitelist the published IP ranges for OpenAI, Anthropic, Perplexity.

Different canonical per UA

Canonical drift confuses agent dedupe. Audit any UA-conditional canonical rules; the canonical should be identical across all UAs.

Linked rubric checks

journey.primary_probe
End-to-end agent probe; runs the same multi-UA fetch under a public scan.
journey.primary_probe
robots_txt.allows_known_agents
Whether each AI crawler is explicitly allowed (or at least not disallowed by wildcard).
robots_txt.allows_known_agents
reachability.status
Whether the page returns 2xx to AgentSpeedBot from a public client.
reachability.status

FAQ

Is this the same as the Agent UA tester?

Same backend pipeline, broader UA list. The UA tester runs 6 UAs to focus on cloaking detection. The Crawler Access Matrix runs 10, adding Claude-User, Google-Extended, Googlebot, Bingbot, and Common Crawl, to give a full picture of who can and cannot reach your site.

Why include Googlebot and Bingbot?

Because misconfigured WAFs frequently block them too. If your search visibility tanks, the matrix tells you whether the cause is your robots.txt, your bot manager, or your CDN, without you having to spelunk through logs.

What does "allowed-different" mean?

The agent got a 200 OK but the body content differs from the browser baseline by more than 20% (Jaccard similarity < 0.80). This is usually cloaking, paywall variants, or "render lite for bots" rules. Agents see worse content than humans.

Do you store the responses?

No. Each run is fetched live and discarded. The result page is rendered server-side; nothing persists.

How do redirects show up?

fetchWithHostPolicy follows up to 5 hops. The "redirected" column shows whether any redirect happened; the final URL column shows where it ended up. SSRF-protected, so every redirect is re-validated against the host policy.

Run a full agent-readiness scan

Full scan

Score the entire site against 30 agent-readiness checks across discoverability, readability, structured data, actionability, and performance.

Get the full report for linode.com →