agentspeed.
← all tools
Tool

AI Bot Policy Generator

Pick allow/disallow on three axes (AI search crawl, AI training, live inference) and get a coherent policy across robots.txt rules, the Cloudflare Content-Signal directive, llms.txt header, terms-of-service clauses, and a human-readable policy page. Pure client-side; nothing leaves your browser.

Tool
Policy summary
✓ AI search crawl✗ AI training✓ Live inference / RAG
robots.txtrobots.txt

Paste at the bottom of your existing robots.txt. Agent-specific rules take precedence over wildcards.

# AgentSpeed: generated AI bot policy
# Paste at the bottom of your existing robots.txt.
# Agent-specific rules take precedence over wildcards.

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Claude-User
Allow: /

# Content-Signal directive (Cloudflare extension).
# Compliant agents honour fine-grained per-purpose consent.
Content-Signal: Allow-Crawl-For-Search, Disallow-AI-Training, Allow-Crawl-For-Inference
llms.txt headerllms-policy-header.txt

Drop in at the top of /llms.txt. The HTML comments are agent-readable and document the policy in machine-and-human-friendly form.

# Acme Inc.

<!-- AI bot policy summary -->
<!-- Search-style citation: ALLOWED -->
<!-- AI training: DISALLOWED -->
<!-- Live inference / RAG: ALLOWED -->
<!-- Full policy: see /policies/ai-policy -->

> Replace this header with your tagline; keep the policy comments
> if you want them visible to agents reading /llms.txt.
AI training clauseai-training-clause.md

One-paragraph clause for your terms of service. Replace placeholders with your specifics.

Acme Inc. does not permit the use of content from this site as training data for AI models. We declare this prohibition via robots.txt (User-agent: GPTBot, CCBot, Google-Extended, Disallow: /) and via the Content-Signal: Disallow-AI-Training directive. Use for any training corpus, including derivative or downstream datasets, requires written consent.
AI crawling clauseai-crawling-clause.md

One-paragraph clause for your terms of service.

Acme Inc. declares the following AI crawling policy. AI search-style crawls are permitted (ChatGPT-User, PerplexityBot, ClaudeBot, Allow: /). Live inference / RAG retrieval is permitted (ChatGPT-User, Claude-User, Allow: /). Compliant agents are expected to honour both the per-agent rules in robots.txt and the Content-Signal directive.
Human-readable policy pageai-policy.md

Drop-in markdown for /policies/ai-policy. Linked from your terms and footer.

# AI bot policy: Acme Inc.

This policy explains how AI agents and crawlers may interact with content published on this site. We follow the robots.txt protocol and the Cloudflare Content-Signal extension. Compliant agents are expected to honour both.

## Permitted and prohibited uses

- ✓ AI search-style citation crawls (ChatGPT browsing, Perplexity, ClaudeBot)
- ✗ Use of content as AI training data
- ✓ Live retrieval for inference / RAG (ChatGPT-User, Claude-User)

## Where this policy lives

Our [robots.txt](https://acme.com/robots.txt) and [llms.txt](https://acme.com/llms.txt) reflect this policy. The Content-Signal directive in robots.txt is the canonical machine-readable form.

## Consent and licensing

Use beyond what is permitted above requires written consent. Contact us to discuss licensing.

Questions or licensing inquiries: contact via the form at https://acme.com/contact.

*Generated with the AgentSpeed AI Bot Policy Generator. AgentSpeed is not a law firm; this copy is a template, not legal advice.*
Why this matters for AI agents

Most sites have inconsistent AI crawler policy: a wildcard disallow in robots.txt that blocks search-style citation, a contradictory training clause in the terms of service, and no human-readable summary at all. Agents and lawyers both look for the explicit signal first, and when three sources disagree, they pick the most restrictive one.

This tool generates all four artifacts from one set of choices, so the answer to “what does this site permit?” is the same no matter where an agent or auditor looks.

Common fixes
Wildcard disallow blocks AI search
A blanket User-agent: * disallow prevents ChatGPT-User and PerplexityBot from citing your site. Most sites want the opposite. Generate a per-axis policy and replace the wildcard.
Allow citation, block training
Common stance: yes to ChatGPT-User browsing, no to GPTBot training. Generate the policy, paste robots.txt, link the policy page from your terms.
No Content-Signal directive
Compliant agents read Content-Signal for fine-grained per-purpose consent. The generator emits the matching directive automatically.
Terms of service contradicts robots.txt
When ToS says "no AI training" but robots.txt allows GPTBot, agents and lawyers default to the most restrictive interpretation. Generate both from one source of truth.
No human-readable policy page
Drop the generated /policies/ai-policy markdown into your CMS and link it from the footer. Agents and reporters both look there first.
Linked rubric checks
FAQ
Is the generated robots.txt the only thing I need?
Robots.txt is the canonical machine-readable form. Compliant agents will honour it. The generator also emits a matching llms.txt header, terms-of-service clauses, and a human-readable policy page so the answer is consistent across every surface.
What is Content-Signal?
A Cloudflare-led extension to robots.txt for per-purpose consent: search vs. inference vs. training vs. summarisation. The generator emits one directive that combines your three choices. Compliant agents read both the per-agent rules and the Content-Signal.
Why three axes instead of per-bot toggles?
Per-bot toggles produce inconsistent policy fast (allow GPTBot but disallow CCBot, though both train AI models). The three-axis model groups bots by what they actually do; the generated robots.txt lists each bot explicitly under its purpose.
Is this legal advice?
No. The terms-of-service clauses are templates, not legal advice. Have your counsel review before publishing. The robots.txt and llms.txt outputs are mechanical and ready to ship.
Does this leave my browser?
No. The policy is generated client-side. Nothing is sent to AgentSpeed; nothing is logged. Copy or download and host it yourself.
Run a full agent-readiness scan
Full scan

Score the entire site against 30 agent-readiness checks across discoverability, readability, structured data, actionability, and performance.

Get the full report →