7 access agent(s) blocked: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, PerplexityBot, YouBot. These keep your site out of AI answers. 11 training-data crawler(s) blocked (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Bytespider, CCBot, cohere-ai, Diffbot, Meta-ExternalAgent, FacebookBot), a licensing choice, not scored.
Remediation
Failing: robots_txt.allows_known_agents
robots_txt.allows_known_agentscriticalauto-fix available
Generated fix
Generated fixreplace file→ https://nytimes.com/robots.txt
Your current /robots.txt disallows these AI agents: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, PerplexityBot, YouBot. Replace your robots.txt with the version below to explicitly allow them while keeping any custom rules you've added below the wildcard block.
# robots.txt for nytimes.com # Explicitly allow well-known AI agents that were previously blocked. # Adjust or remove specific agents as your policy requires. User-agent: * Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Perplexity-User Allow: / User-agent: PerplexityBot Allow: / User-agent: YouBot Allow: / Sitemap: https://nytimes.com/sitemap.xml
save as
https://nytimes.com/robots.txt- ›This snippet replaces your existing robots.txt. Merge any custom Disallow rules you have today below the wildcard block.
- ›Cloudflare and a few CDNs cache robots.txt aggressively, so purge the cache after deploy to confirm AI agents see the change.
Why and how