Slow server response (high TTFB)

Your server takes too long to return the first byte, so agents time out before reading your page.

Time to First Byte (TTFB) is the gap between when an agent makes a request and when your server starts sending back HTML. AI crawlers run with strict budgets, typically 3–8 seconds total per page. If your TTFB is 4 seconds, the agent has roughly 0–4 seconds left to download, parse, and move on; many will abandon the request entirely. "Slow server response" usually means TTFB > 1.5s, full-render > 5s, or payload > 1MB, any of which can cause an agent to drop your page from its index.

Agents are not patient browsers. They run thousands of fetches in parallel, and a slow page just gets retried less often (or skipped). The cost compounds: a slow page is fetched less often → updates take longer to propagate → cited content goes stale → the agent loses confidence and stops citing the domain. Fast TTFB is the prerequisite for everything else on this list; the best llms.txt in the world does not help if half your pages time out.

Pages with TTFB > 1s are excluded from many real-time agent surfaces (ChatGPT browse, Perplexity live results) because they blow past the per-fetch budget. Slow pages also degrade SEO (Core Web Vitals), increase hosting costs (queued requests cost CPU), and hurt human conversion. Speeding up the server is one of the highest-leverage performance fixes: once-and-done, applies to every page on the site, and the gains stack with caching and CDN strategy.

Run `curl -o /dev/null -s -w 'TTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n' https://your-domain.com/`. TTFB above 1 second is concerning, above 2 seconds is a problem. Or open DevTools → Network → click the document request → Timing tab; look at "Waiting for server response (TTFB)". WebPageTest and PageSpeed Insights will both surface the number alongside diagnostics.

TTFB has three causes, ranked by frequency: (1) server-side rendering work that recomputes per request, fixed with caching (full-page cache, ISR, edge cache); (2) database queries on the hot path, fixed with query caching, indexes, and N+1 elimination; (3) origin geographically far from agents, fixed by putting a CDN in front of the origin so agent fetches hit edge POPs instead of your origin every time. Address (1) first; it is usually the biggest win.

# HTTP response headers for a public, cacheable page.
# CDNs and agent caches read these to decide whether to refetch.

Cache-Control: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400
ETag: "v3-2025-04-12"
Vary: Accept-Encoding

# max-age=300        — browsers/agents reuse for 5 min
# s-maxage=3600      — shared caches (CDN) reuse for 1 hour
# stale-while-revalidate=86400 — serve stale for up to 1 day while
#   refetching in the background; near-zero TTFB on cache miss

The reference snippet above is platform-agnostic. Below are the concrete steps for the four most common stacks plus a generic fallback for any other host.

import { unstable_cache } from 'next/cache';
import { getProductFromDb } from '@/lib/products';

// Regenerate the page in the background at most once a minute.
export const revalidate = 60;

// Cache the DB lookup separately so it can be reused across requests
// even when the page is revalidating.
const getProduct = unstable_cache(
  (slug: string) => getProductFromDb(slug),
  ['product'],
  { revalidate: 60, tags: ['products'] },
);

export default async function ProductPage({
  params,
}: {
  params: Promise<{ slug: string }>;
}) {
  const { slug } = await params;
  const product = await getProduct(slug);
  return <article>{/* … */}</article>;
}

# Tell Cloudflare / a downstream CDN that anonymous HTML is cacheable.
<IfModule mod_headers.c>
  <FilesMatch "\.(html|htm)$">
    Header set Cache-Control "public, max-age=300, s-maxage=3600, stale-while-revalidate=86400"
  </FilesMatch>
</IfModule>

# Bypass cache for admin / preview / cart cookies.
<IfModule mod_rewrite.c>
  RewriteCond %{HTTP_COOKIE} (wordpress_logged_in_|wp-postpass_|comment_author_) [NC]
  RewriteRule .* - [E=NO_CACHE:1]
  Header set Cache-Control "private, no-store" env=NO_CACHE
</IfModule>

{%- comment -%}
  Bad: iterates every product in the collection at render time.

  {% for product in collections.all.products %}
    <h3>{{ product.title }}</h3>
  {% endfor %}

  Good: explicit limit. Renders fast even for large catalogues.
{%- endcomment -%}
{% for product in collections.featured.products limit: 12 %}
  <a href="{{ product.url }}">
    <h3>{{ product.title }}</h3>
    <p>{{ product.price | money }}</p>
  </a>
{% endfor %}

<!-- Slow analytics or chat widgets go here, async, with defer.
     They run after the main HTML has rendered, so agent TTFB stays
     fast and the widgets load when the browser has spare cycles. -->
<script async defer src="https://cdn.example.com/widget.js"></script>

[[headers]]
  for = "/*"
  [headers.values]
    Cache-Control = "public, max-age=300, s-maxage=3600, stale-while-revalidate=86400"

# Long-lived cache for fingerprinted assets (Next.js _next/static, etc.)
[[headers]]
  for = "/_next/static/*"
  [headers.values]
    Cache-Control = "public, max-age=31536000, immutable"

• Setting Cache-Control: no-cache or no-store everywhere "to be safe." This forces every agent to re-fetch from origin every time, defeating CDN caching entirely. Use private no-store only for genuinely sensitive responses.
• Caching the full HTML but not the API responses behind it. The page is fast, but every fetch triggers fresh DB queries; under load, origin slows down anyway. Cache the data layer separately.
• Geographic mismatch: origin in us-east-1 but agents fetching from Europe and Asia. A CDN solves this in 30 minutes.
• Synchronous third-party scripts in <head> ("for tracking"). They block the parser; if the script is slow, your TTFB-equivalent for downstream parsers (including agents) suffers. Use async / defer.
• Optimising payload (gzip, minification) before fixing the TTFB. Payload optimisations help on slow networks but don't reduce server-side waiting. Fix TTFB first, payload second.

Re-run `curl -w 'TTFB: %{time_starttransfer}s\n' https://your-domain.com/` from at least two regions (use a remote VM if your laptop is on the same continent as origin). Aim for sub-300ms TTFB on cached pages, sub-1s on uncached. Run a fresh AgentSpeed scan, and TTFB and full-render checks should pass. PageSpeed Insights and WebPageTest will corroborate.