Rubric / Discoverability
robots.txt present
robots_txt.present
weight 1deterministic
A reachable robots.txt exists at the site root.
Definition
Check ID
robots_txt.present
Category
Discoverability
Weight
1
Counts toward the score
Yes.
Evaluated by
A fixed rule with no threshold of our choosing. The same input always produces the same verdict.
Standard maturity
established
Canonical spec
Introduced in
r2026.04.1
Retired in
Still active.
History
Every change to the instrument— including the versions that changed this check
Machine-readable
/rubric.json— this check, and every other, as data
How often this check is right
No rate published. Too few labelled cases to publish a rate: 1 flagged case (30 needed to publish), 1 defective case (30 needed to publish). The counts are the honest answer here. The matrix behind robots_txt.present is 5 labelled cases — reported here rather than turned into a percentage that would read as more certain than the evidence is.
Corpus v1, measured under rubric r2026.11.0. Full method and every other check at /rubric/accuracy.
How to fix it
Add a `/robots.txt` file at the site root. Even an allow-all file signals intent to agents.
Does your site pass this check?
Free scan, 30 seconds, no signup — this check and the other 35, with the exact evidence for each verdict.
Scan my site →