petrolvibes.com

robots.txt is a small text file at yoursite.com/robots.txt where you tell visiting robots what they may read. Many sites carry an old rule that blocks ALL robots, which now also blocks the AI assistants your customers ask for recommendations.

ChatGPT, Claude, and Perplexity each send a named crawler (GPTBot, ClaudeBot, PerplexityBot). If your robots.txt turns them away, they cannot read your pages. So when a customer asks an assistant about your product, the assistant answers from what everyone ELSE says about you, or recommends a competitor it could read.

1. Shopify: Edit robots.txt.liquid and remove any Disallow lines that match GPTBot, ClaudeBot, PerplexityBot, etc.
2. Open yoursite.com/robots.txt in a browser to see what you have today.
3. Look for "User-agent: *" followed by "Disallow: /". That single pair blocks everything, AI agents included.
4. Add an explicit allow block for each AI agent you want (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot), placed after the wildcard rule so it wins.
5. The generated robots.txt in the Fix it section below has the exact lines for this site. Copy it, or merge the blocks into your existing file.

• After deploying, open yoursite.com/robots.txt and confirm the new blocks are live, not cached.
• Hit Re-scan above: "robots.txt allows AI agents" should flip to PASS.
• The free robots.txt checker at /tools/robots-txt-checker shows per-agent allow/block for your live file.

• Don’t delete rules you added deliberately. If you block a specific scraper for a reason, keep that block; just don’t let a blanket rule catch the assistants too.
• Allowing crawlers you actually want blocked just to raise a score is backwards. The score measures readiness for agents you WANT; decide the policy first, then express it precisely.
• Don’t serve a different robots.txt to different visitors. Inconsistent answers get you treated as unreliable by every crawler.

Of everything your server sends for this page, very little is actual readable text. The rest is code, styling, and markup wrapper. Agents fetched a lot of bytes and found few words.

Agents work with retrieval budgets. A page that is 2% text either gets skimmed (and mis-summarised) or skipped. More signal per byte means more of your actual message survives into the agent’s summary.

1. Confirm your real copy is server-rendered. See the JavaScript check; these two usually fail together.
2. Cut boilerplate wrappers: deeply nested divs, inline SVG logos repeated per section, base64 images inlined into HTML.
3. Move large inline styles and scripts into external files.
4. Check how much of the document is inline script: your report now says. If most of it is, that is your framework serializing state for hydration, often a second copy of text already in the HTML. Send only the props a component cannot recompute, and render statically where a page does not need to hydrate.

• Re-scan: "Text-to-markup ratio" improves, and "Page weight" often improves alongside.

• Don’t pad the page with keyword text to inflate the ratio. The ratio is a proxy for substance, and stuffing is the opposite of substance.

The meta description is a one-sentence summary of the page that you control, embedded in the HTML head. Without it, machines compose their own summary from whatever text appears first, often cookie-notice or navigation boilerplate.

This is the sentence you get to write about yourself. Pages without one are summarised by autopilot, and the autopilot frequently leads with the least representative text on the page.

1. Write one honest sentence per page (120–160 characters) saying what the page offers.
2. Add it as <meta name="description" content="…"> per page via your framework or SEO plugin.

• Re-scan: "Meta description" flips to PASS.

• Don’t reuse one description site-wide. A shared summary describes none of the pages.
• Describe the page, not your aspirations. Agents compare the description against the content and discount pages where they diverge.

Your structured data exists but only covers the basics (say, Organization) while the page is really a product page, an article, or an FAQ. The machine-readable caption is missing the part that describes what this page actually is.

An agent looking for "running shoes under $100" matches against Product markup with offers. A product page carrying only Organization markup is invisible to that query shape: the agent knows who you are but not what this page sells.

1. Identify the page’s real type: Product (with offers), Article, FAQPage, HowTo, LocalBusiness, Event.
2. Add that type’s JSON-LD alongside the Organization block, filled from the page’s actual content.
3. For Product: include name, price, priceCurrency, and availability. Those are the fields agents actually query.

• Re-scan: "Schema type coverage" improves. Keep "Structured data validates" green while you add.
• The coherence checks below compare your declared price and name to the visible page; they should stay (or flip to) PASS.

• Only declare types the page genuinely is. A blog post marked up as a Product to appear in shopping queries is the exact incoherence assistants are getting better at detecting.
• Keep declared values synced with the page. If a sale changes the visible price, the JSON-LD price must change in the same deploy.

HTTP responses can carry a Link header: the same canonical and alternate-language information your HTML declares, but delivered in the response envelope itself. Agents that only send a HEAD request, or that decide what to do while the page is still streaming, read the header without parsing any HTML. Yours is missing (or carries none of the relations agents use: canonical, alternate, describedby).

An agent triaging fifty URLs doesn’t want to download and parse fifty pages to learn which are duplicates of which. The Link header answers at the cheapest possible layer. Sites that provide it get correctly de-duplicated and correctly language-routed even by the most minimal fetchers.

1. Emit a Link header alongside each page, mirroring what your HTML head already declares. Example: Link: <https://yoursite.com/page>; rel="canonical".
2. Most stacks set this in one place: Next.js headers() config, an nginx add_header line, or a CDN response-header rule.
3. If you serve translations, add rel="alternate" entries with hreflang, matching your HTML.

• Run: curl -sI https://yoursite.com | grep -i "^link:" and confirm the relations appear.
• Hit Re-scan above: "Link response headers" flips to PASS.

• The header must AGREE with the HTML. A Link header pointing one place while the in-page canonical points another gives machines two contradictory answers, which is worse than one missing answer.
• Don’t inject the same site-wide canonical on every page via a blanket CDN rule; each page names its own clean URL, exactly as in the HTML tag.