Rubric / Discoverability

robots.txt allows AI agents

robots_txt.allows_known_agents

weight 2deterministic

robots.txt does not disallow answer-time AI access agents (ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, YouBot, Amazonbot). Training-data crawler blocks (GPTBot, CCBot, Google-Extended, …) are reported as informational and never fail the check.

Definition

Check ID

robots_txt.allows_known_agents

Category

Discoverability

Weight

Counts toward the score

Yes.

Evaluated by

A fixed rule with no threshold of our choosing. The same input always produces the same verdict.

Standard maturity

established

Canonical spec

https://datatracker.ietf.org/doc/html/rfc9309

Introduced in

r2026.04.1

Retired in

Still active.

History

Every change to the instrument— including the versions that changed this check

Machine-readable

/rubric.json— this check, and every other, as data

How often this check is right

No rate published. Too few labelled cases to publish a rate: 1 flagged case (30 needed to publish), 1 defective case (30 needed to publish). The counts are the honest answer here. The matrix behind robots_txt.allows_known_agents is 3 labelled cases — reported here rather than turned into a percentage that would read as more certain than the evidence is.

Corpus v1, measured under rubric r2026.11.0. Full method and every other check at /rubric/accuracy.

How to fix it

If you want your content in AI answers, explicitly allow the access agents in robots.txt — blocking them keeps assistants from reading or citing your site at answer time. Blocking training-data crawlers (GPTBot, CCBot, Google-Extended) is a separate licensing choice and does not affect this check.

Does your site pass this check?

Free scan, 30 seconds, no signup — this check and the other 35, with the exact evidence for each verdict.

Scan my site →