sncf-connect.com

Your site has no robots.txt file at all. That file is the front-door sign for every robot: what to read, what to skip, and where your sitemap lives.

Without one, every crawler guesses. Most treat a missing file as "allowed", but you lose the one place to point them at your sitemap and to state your policy explicitly. An explicit welcome is a clearer signal than silence.

1. Create a plain text file named robots.txt.
2. Start permissive: "User-agent: *" then "Allow: /", plus a "Sitemap:" line pointing at your sitemap URL.
3. Serve it at exactly yoursite.com/robots.txt (site root, not a subfolder).

• Open yoursite.com/robots.txt. You should see your file, not a 404 page.
• Re-scan: "robots.txt present" flips to PASS, and if you added the Sitemap line, "Sitemap present" often flips with it.

• Don’t copy a robots.txt from another site. Theirs encodes their policy, including blocks you may not want.
• Don’t return an HTML error page at that URL with status 200; robots parsers read it as garbage rules.

A sitemap is a machine-readable list (sitemap.xml) of every page on your site. It is how crawlers find pages that aren’t linked from your homepage.

Agents discovering your site crawl outward from the pages they know. Without a sitemap, anything not linked from a top-level page (docs, older articles, product variants) may simply never be read, so it can’t be quoted or recommended.

1. Most platforms generate one automatically: Next.js (app/sitemap.ts), WordPress (Yoast/Rank Math), Shopify and Wix (built in). Turn it on rather than hand-writing.
2. Add a "Sitemap: https://yoursite.com/sitemap.xml" line to robots.txt so crawlers find it.
3. Submit it once in Google Search Console and Bing Webmaster Tools for the search side.

• Open yoursite.com/sitemap.xml. It should list real page URLs, not error.
• Re-scan: "Sitemap present" flips to PASS.

• Don’t list URLs that redirect or 404. A sitemap full of dead links teaches crawlers to distrust it.
• Don’t include private or staging URLs; a sitemap is a public document.

Content-Signal is a newer robots.txt directive (pushed by Cloudflare) that lets you state a granular AI policy: for example "don’t train on my content, but summarising it in answers is fine". Classic robots.txt only offers all-or-nothing per crawler; this adds the middle ground. We looked for a Content-Signal line in your robots.txt and response headers and found none.

Without a granular signal, crawlers infer your intent from blunt allow/block rules, and sites often block more than they mean to just to avoid training use. A declared signal lets you keep answer visibility (summaries, citations) while opting out of what you object to. This is an emerging standard: honoring is voluntary and adoption is early, which is why it is flagged as emerging and weighs little.

1. Decide your actual policy first: are you fine with AI training on your content? With summarisation in answers? With search indexing?
2. Express it as one line in robots.txt, for example: Content-Signal: ai-train=no, ai-summarize=yes.
3. Keep your existing User-agent rules; the signal adds nuance on top, it doesn’t replace them.

• Open yoursite.com/robots.txt and confirm the Content-Signal line is live.
• Hit Re-scan above: "Content-Signal directives" flips to PASS.

• Don’t declare signals that contradict your User-agent rules, such as ai-summarize=yes while blocking every AI crawler; conflicting instructions get you treated as unreliable.
• Don’t add the line just for the score. It is a public policy statement; say what you mean, because compliant crawlers will act on it.

MCP (Model Context Protocol) is how AI assistants call tools. A server card is a small JSON file at /.well-known/mcp/server-card.json that announces "this site offers these tools" so an assistant can find them without scraping your pages. We requested that file from your site and found none.

Assistants that support MCP look for the card at that exact address. With one, "book an appointment on yoursite.com" can become a tool call instead of screen-driving your UI. This is an emerging standard, so it weighs little today; publishing early costs one static file and makes you visible to the assistants adopting it.

1. If you already run an MCP server, describe it: name, description, and endpoint URL in the card.
2. If you don’t, a minimal truthful card naming your site and linking your API docs is still valid. The generated starter in the Fix it section below is filled in for this site.
3. Serve it at exactly /.well-known/mcp/server-card.json with content type application/json.

• Open yoursite.com/.well-known/mcp/server-card.json and confirm it returns your JSON, not a 404 page.
• Hit Re-scan above: "MCP server card" flips to PASS.

• Don’t advertise tools that don’t exist. An assistant that calls a declared endpoint and gets a 404 learns to skip your card entirely.
• Don’t let your framework rewrite /.well-known/* to your single-page app; the file must return raw JSON, not your HTML shell.