Remediation

A reachable robots.txt exists at the site root.

robots_txt.presentdiscoverabilityweight 1deterministicauto-fix available

Why it matters

Underlying specification: https://datatracker.ietf.org/doc/html/rfc9309

How to fix

See the full check definition in the rubric (linked above) for the canonical reference.
Read the underlying specification at https://datatracker.ietf.org/doc/html/rfc9309.

Generated snippet

The snippet below uses your-domain.example as a placeholder. Replace it with your actual host before deploying. For a fix tailored to your live site, run a scan and open the per-domain remediation page.

Generated fixnew file→ https://your-domain.example/robots.txt

AgentSpeed couldn't find a /robots.txt at the site root. Even an allow-all file signals intent to AI agents and unblocks the rest of the discoverability checks (sitemap, llms.txt, content signals).

# robots.txt for your-domain.example
# Allow all crawlers (including AI agents) by default.

User-agent: *
Allow: /

Sitemap: https://your-domain.example/sitemap.xml

save as https://your-domain.example/robots.txt

›Save this exactly as `/robots.txt` at the apex (not under /public, /static, etc.).
›Make sure the file is served as `text/plain` with a 200 status. Many static hosts default to HTML rendering for unknown paths.
›If you intentionally want to block specific agents, add a `User-agent: <name>` block with `Disallow: /` instead of removing this file.

Hand it to your AI assistant

Paste this into Claude Code, Cursor, or any assistant with your repository open. It carries the steps, how to verify them, and the shortcuts to avoid. It is generic, though: run a scan to get one written from what your pages actually do.

Run a scan to check your site

Verify on your site

Run a scan against your domain to see whether robots_txt.present currently passes, warns, or fails. Free, no signup; the score page is permanent and citable.

Run a scan →See the full rubric →