Remediation

A reachable robots.txt exists at the site root.

robots_txt.presentdiscoverabilityweight 1deterministicauto-fix available

Why it matters

Underlying specification: https://datatracker.ietf.org/doc/html/rfc9309

How to fix

See the full check definition in the rubric (linked above) for the canonical reference.
Read the underlying specification at https://datatracker.ietf.org/doc/html/rfc9309.

Generated snippet

The snippet below uses your-domain.example as a placeholder. Replace it with your actual host before deploying. For a fix tailored to your live site, run a scan and open the per-domain remediation page.

Generated fixnew file→ https://your-domain.example/robots.txt

AgentSpeed couldn't find a /robots.txt at the site root. Even an allow-all file signals intent to AI agents and unblocks the rest of the discoverability checks (sitemap, llms.txt, content signals).

# robots.txt for your-domain.example
# Allow all crawlers (including AI agents) by default.

User-agent: *
Allow: /

Sitemap: https://your-domain.example/sitemap.xml

save as https://your-domain.example/robots.txt

›Save this exactly as `/robots.txt` at the apex (not under /public, /static, etc.).
›Make sure the file is served as `text/plain` with a 200 status. Many static hosts default to HTML rendering for unknown paths.
›If you intentionally want to block specific agents, add a `User-agent: <name>` block with `Disallow: /` instead of removing this file.

Run a scan to check your site

Verify on your site

Run a scan against your domain to see whether robots_txt.present currently passes, warns, or fails. Free, no signup; the score page is permanent and citable.

Run a scan →See the full rubric →