/robots.txt is the original opt-in/out protocol for web crawlers. The same protocol governs AI agents. The mistake most sites make is inheriting a wildcard User-agent: * Disallow rule into AI traffic without realising. ChatGPT, Claude, and Perplexity respect explicit agent rules first; if you’ve blocked *, you’ve blocked them too.
Three precedence rules
- Agent-specific rules win. A
User-agent: GPTBotAllow overrides a wildcard Disallow. - Most-specific path wins. Within an agent’s rule block, longer path patterns take precedence.
- Allow beats Disallow on a tie. When two rules of equal specificity conflict, the Allow wins.
Recommended pattern
Allow named AI agents you want crawling your site. The cheapest single change for AI visibility is appending agent-specific blocks to robots.txt:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Place these AFTER any wildcard * rule so they take precedence. To verify, run any URL through the AgentSpeed robots.txt checker; it returns paste-ready fix snippets when known agents are blocked.
What about Bytespider?
Bytespider is ByteDance’s crawler. It has nothing to do with Googlebot or other search crawlers. You can block one without affecting the other. That’s exactly what agent-specific rules are for.
What is Content-Signal?
Content-Signal is a Cloudflare-led extension to robots.txt that lets you grant fine-grained consent: search vs. inference vs. training vs. summarisation. The directive is per-purpose: Allow-Crawl-For-Search, Disallow-Crawl-For-Inference. Compliant agents honour it. AgentSpeed surfaces parsed Content- Signal directives in the robots.txt checker.