Your current /robots.txt disallows this AI agent: PerplexityBot. Replace your robots.txt with the version below to explicitly allow them while keeping any custom rules you've added below the wildcard block.
# robots.txt for indianexpress.com
# Explicitly allow well-known AI agents that were previously blocked.
# Adjust or remove specific agents as your policy requires.
User-agent: *
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://indianexpress.com/sitemap.xml
save as https://indianexpress.com/robots.txt
›This snippet replaces your existing robots.txt. Merge any custom Disallow rules you have today below the wildcard block.
›Cloudflare and a few CDNs cache robots.txt aggressively, so purge the cache after deploy to confirm AI agents see the change.
Why and how
1 access agent(s) blocked: PerplexityBot. These keep your site out of AI answers. 7 training-data crawler(s) blocked (ClaudeBot, anthropic-ai, Applebot-Extended, Bytespider, cohere-ai, Diffbot, Meta-ExternalAgent), a licensing choice, not scored.