AgentSpeedBot

AgentSpeedBot is a public-benefit scanner that reads publicly reachable web pages and grades how well they perform for AI agents. We honor robots.txt for our own user-agent. We do not crawl your site: one scan fetches the landing page plus a fixed set of standard machine-readable resources (robots.txt, sitemap.xml, and the /.well-known agent-protocol endpoints), around a dozen requests, once.

AgentSpeedBot/1.0 (+https://agentspeed.com/bot)

Our own scheduled crawling is capped by a hard, fail-closed budget of 30 scans per hour across the entire corpus, and each tracked domain is scanned at most once every 30 days. If the budget counter is ever unreachable, the crawler stops rather than running unmetered.

User-agent: AgentSpeedBot
Allow: /

User-agent: AgentSpeedBot
Disallow: /

When the next scan sees this rule, we automatically set your domain to publicOptOut and the page at agentspeed.com/score/<your-domain> is replaced with a “removed at owner request” notice, and your domain leaves the crawl rotation permanently. Removing the rule and submitting your domain on the homepage restores the score.

If you can’t edit robots.txt (CDN, locked-down infra, third-party host), email bot@agentspeed.com from an address at the domain in question. We will verify ownership and remove the public page within one business day.

bot@agentspeed.com for abuse, false positives, score disputes, and IP-range inquiries. A complaint pauses crawling of your domain first and gets investigated second.