Missing XML sitemap

No /sitemap.xml means agents and search engines must discover every page through link-following alone.

An XML sitemap is a machine-readable list of every public URL on your site, served at /sitemap.xml as application/xml. Crawlers (both search engines and AI agents) fetch it as one of their first requests on a new domain. Without it, the crawler has to follow links from your homepage and may miss pages that aren't linked from the navigation (older blog posts, deep product pages, archived content).

Agents have a finite per-domain crawl budget. With a sitemap, they fetch a curated list and prioritise; without one, they spend their budget on whatever the homepage links to and stop. Recently published pages, paginated content, and deep product catalogues are the most common casualties. A sitemap also carries lastmod timestamps, which signal to agents which pages are fresh and worth re-fetching.

Pages that are not in the sitemap are not in the index. Pages that are not in the index are not cited by agents. For e-commerce: every product not in the sitemap is invisible to AI shopping agents. For content sites: every article not in the sitemap is invisible to answer engines. The fix is mechanical, ships once, and updates automatically thereafter.

Open https://your-domain.com/sitemap.xml directly. Expect HTTP 200 with content-type application/xml and a body that opens with <?xml version="1.0" ...?> and contains <urlset> wrapping <url><loc> entries. If you get a 404, you have no sitemap. If you get HTML, your server is misrouting. Also check robots.txt for a 'Sitemap:' directive, since search engines and agents check there too.

Generate sitemap.xml at build time (preferred) or as a route handler that queries your CMS. Each <url> entry needs a <loc> (absolute URL) and ideally a <lastmod> (ISO 8601 timestamp). Reference the sitemap from robots.txt with a 'Sitemap:' line. Keep individual sitemaps under 50 MB / 50,000 URLs; use a sitemap index for larger sites.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://acme.com/</loc>
    <lastmod>2026-04-15T10:00:00Z</lastmod>
  </url>
  <url>
    <loc>https://acme.com/products/skillet-10</loc>
    <lastmod>2026-04-12T08:30:00Z</lastmod>
  </url>
  <url>
    <loc>https://acme.com/blog/seasoning-guide</loc>
    <lastmod>2026-04-08T14:15:00Z</lastmod>
  </url>
  <url>
    <loc>https://acme.com/pricing</loc>
    <lastmod>2026-03-30T09:00:00Z</lastmod>
  </url>
</urlset>

The reference snippet above is platform-agnostic. Below are the concrete steps for the four most common stacks plus a generic fallback for any other host.

import type { MetadataRoute } from 'next';
import { getAllProducts } from '@/lib/products';
import { getAllPosts } from '@/lib/posts';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const base = 'https://acme.com';
  const products = await getAllProducts();
  const posts = await getAllPosts();

  return [
    { url: base, lastModified: new Date() },
    { url: `${base}/pricing`, lastModified: new Date() },
    ...products.map((p) => ({
      url: `${base}/products/${p.slug}`,
      lastModified: p.updatedAt,
    })),
    ...posts.map((p) => ({
      url: `${base}/blog/${p.slug}`,
      lastModified: p.publishedAt,
    })),
  ];
}

# Alias /sitemap.xml to WordPress's core sitemap so agents that hard-code
# the conventional path also find it.
RewriteRule ^sitemap\.xml$ /wp-sitemap.xml [L,R=301]

# Shopify's default robots.txt.liquid already contains:
Sitemap: {{ shop.url }}/sitemap.xml

# If you've overridden robots.txt.liquid, make sure this line is preserved.

1. Site settings → SEO tab
2. Toggle "Auto-generate sitemap" → on
3. Publish
4. Confirm https://your-domain.com/sitemap.xml returns 200

import { readdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';

const BASE = 'https://acme.com';
const PAGES_DIR = './src/pages';

const files = await readdir(PAGES_DIR, { recursive: true });
const urls = files
  .filter((f) => f.endsWith('.html'))
  .map((f) => f.replace(/\.html$/, '').replace(/\/index$/, ''))
  .map((slug) => `  <url><loc>${BASE}/${slug}</loc></url>`);

const xml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${urls.join('\n')}
</urlset>`;

await writeFile(join('./public', 'sitemap.xml'), xml);

• Listing relative URLs (/about) instead of absolute URLs (https://acme.com/about). The XML schema requires absolute. Relative URLs are silently dropped by most parsers.
• Including non-canonical URLs, e.g. both /products/foo and /products/foo?ref=email. Pick the canonical and list only that. Tracking-parameter URLs in the sitemap can cause crawl-budget waste and duplicate-content signals.
• Forgetting to update lastmod. Agents use lastmod to decide whether to refetch; static lastmod=2024-01-01 across the whole site means agents stop checking for updates.
• Not adding the Sitemap: line to robots.txt. Even if the sitemap exists at /sitemap.xml, naming it explicitly in robots.txt removes ambiguity.
• Including pages with noindex meta tags. The sitemap and the meta tag contradict each other; agents interpret it as a configuration mistake and may drop the URL.

Fetch https://your-domain.com/sitemap.xml and confirm it returns 200 with valid XML. Spot-check three URLs from the body to make sure they load. Submit the sitemap to Google Search Console and Bing Webmaster Tools (one-time). Then run a fresh AgentSpeed scan, and the "sitemap present" check should pass.