Missing llms.txt

Your site has no /llms.txt file, so AI agents have to guess which pages matter most.

llms.txt is a plain-text manifest at the root of your site (https://example.com/llms.txt) that lists the canonical URLs you most want AI agents to read. When it is missing, agents fall back to crawling your sitemap, your homepage, and whatever links they happen to find, which often means they index marketing copy and ignore your docs, pricing, and product pages.

AI agents have small attention budgets. ChatGPT, Claude, Perplexity and emerging shopping agents typically fetch only a handful of pages per domain before answering a user. llms.txt lets you put your best content first: the pages that explain what you do, what you charge, and how to take action. Without it, agents read whatever a crawler happened to surface, which is rarely the page you would have chosen.

When agents cite competitors instead of you in answers, you lose the impression entirely. There is no second-page result to recover from. Sites with a curated llms.txt see meaningfully higher citation rates in agent answers (because the agent ingested the right pages) and faster onboarding for AI shopping agents that need to identify your primary offering. Cost to add: ~15 minutes. Cost of not adding: every agent query that picks the wrong page or no page at all.

Open https://your-domain.com/llms.txt in a browser. If you see a 404, the file is missing. If you see HTML (your homepage), the file is missing and your server is misrouting. The file should return HTTP 200 with content-type text/plain and a body that starts with "# Site Name" followed by markdown-style sections.

Create a plain-text file at the root of your site called llms.txt. Open it with "# <Your site name>", a one-paragraph description of what you do, then sections (## Docs, ## Pricing, ## Blog, …) listing the canonical URLs you most want agents to read. Keep it under 50 KB, since long manifests get truncated.

# Acme

> Acme sells durable cookware to home cooks. We ship to the US and Canada.

## Docs

- [Quick start](https://acme.com/docs/quick-start): How to use your first pan.
- [Care guide](https://acme.com/docs/care): Cleaning, seasoning, longevity.

## Pricing

- [Pricing](https://acme.com/pricing): Our four product tiers and what each includes.

## Company

- [About](https://acme.com/about): Founding story, materials, manufacturing.
- [Contact](https://acme.com/contact): Email and support hours.

The reference snippet above is platform-agnostic. Below are the concrete steps for the four most common stacks plus a generic fallback for any other host.

export const dynamic = 'force-static';

const BODY = `# Acme

> Acme sells durable cookware to home cooks.

## Docs
- [Quick start](https://acme.com/docs/quick-start)
- [Care guide](https://acme.com/docs/care)

## Pricing
- [Pricing](https://acme.com/pricing)
`;

export function GET() {
  return new Response(BODY, {
    headers: { 'content-type': 'text/plain; charset=utf-8' },
  });
}

# Place above the WordPress rewrite block.
<Files "llms.txt">
  Require all granted
</Files>
RewriteRule ^llms\.txt$ - [L]

{% layout none %}# {{ shop.name }}

> {{ shop.description }}

## Products
{% for product in collections.all.products limit: 25 %}
- [{{ product.title }}](https://{{ shop.domain }}/products/{{ product.handle }})
{% endfor %}

const BODY = `# Acme

> Acme sells durable cookware.

## Pages
- [Home](https://acme.com/)
- [Pricing](https://acme.com/pricing)
`;

export default {
  async fetch(req) {
    const { pathname } = new URL(req.url);
    if (pathname === '/llms.txt') {
      return new Response(BODY, {
        headers: { 'content-type': 'text/plain; charset=utf-8' },
      });
    }
    return fetch(req);
  },
};

[[headers]]
  for = "/llms.txt"
  [headers.values]
    Content-Type = "text/plain; charset=utf-8"
    Cache-Control = "public, max-age=3600"

• Returning HTML or a 200 page that says "Not Found." Agents follow the HTTP status, not the body. The file must be a real 404 or a real 200 with text/plain.
• Putting llms.txt under /docs/ or /public/ instead of the site root. Agents only check the apex.
• Listing every URL on the site. The manifest is a curated guide, not a sitemap. 20–50 high-value links beats 500 noisy ones.
• Embedding rich HTML or JavaScript. The format is plain text with markdown-style links, and anything else gets stripped.
• Setting cache-control to no-store, which forces every agent to re-fetch on every query. 1 hour is the sweet spot.

After deploying, visit https://your-domain.com/llms.txt in a private browser window. It should render as plain text and the network tab should show 200 with content-type text/plain. Then run a fresh AgentSpeed scan. The "llms.txt present" check should flip from fail to pass within ~30 seconds; the overall score will tick up by the check's rubric weight.