uliceistorije.rs

robots.txt is a small text file at yoursite.com/robots.txt where you tell visiting robots what they may read. Many sites carry an old rule that blocks ALL robots, which now also blocks the AI assistants your customers ask for recommendations.

ChatGPT, Claude, and Perplexity each send a named crawler (GPTBot, ClaudeBot, PerplexityBot). If your robots.txt turns them away, they cannot read your pages. So when a customer asks an assistant about your product, the assistant answers from what everyone ELSE says about you, or recommends a competitor it could read.

1. Open yoursite.com/robots.txt in a browser to see what you have today.
2. Look for "User-agent: *" followed by "Disallow: /". That single pair blocks everything, AI agents included.
3. Add an explicit allow block for each AI agent you want (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot), placed after the wildcard rule so it wins.
4. The generated robots.txt in the Fix it section below has the exact lines for this site. Copy it, or merge the blocks into your existing file.

• After deploying, open yoursite.com/robots.txt and confirm the new blocks are live, not cached.
• Hit Re-scan above: "robots.txt allows AI agents" should flip to PASS.
• The free robots.txt checker at /tools/robots-txt-checker shows per-agent allow/block for your live file.

• Don’t delete rules you added deliberately. If you block a specific scraper for a reason, keep that block; just don’t let a blanket rule catch the assistants too.
• Allowing crawlers you actually want blocked just to raise a score is backwards. The score measures readiness for agents you WANT; decide the policy first, then express it precisely.
• Don’t serve a different robots.txt to different visitors. Inconsistent answers get you treated as unreliable by every crawler.

Of everything your server sends for this page, very little is actual readable text. The rest is code, styling, and markup wrapper. Agents fetched a lot of bytes and found few words.

Agents work with retrieval budgets. A page that is 2% text either gets skimmed (and mis-summarised) or skipped. More signal per byte means more of your actual message survives into the agent’s summary.

1. Confirm your real copy is server-rendered. See the JavaScript check; these two usually fail together.
2. Cut boilerplate wrappers: deeply nested divs, inline SVG logos repeated per section, base64 images inlined into HTML.
3. Move large inline styles and scripts into external files.
4. Check how much of the document is inline script: your report now says. If most of it is, that is your framework serializing state for hydration, often a second copy of text already in the HTML. Send only the props a component cannot recompute, and render statically where a page does not need to hydrate.

• Re-scan: "Text-to-markup ratio" improves, and "Page weight" often improves alongside.

• Don’t pad the page with keyword text to inflate the ratio. The ratio is a proxy for substance, and stuffing is the opposite of substance.

Time to first byte is how long your server thinks before it starts answering: the gap between an agent asking for the page and the first byte of HTML arriving. Yours crossed our threshold (under 200ms scores full marks, over 2 seconds fails).

Agents fetch with tight timeouts and many pages in parallel. A server that takes seconds to start answering gets cut off mid-queue, and a page that times out is a page that was never read. Speed here is not polish; it decides whether the content gets fetched at all.

1. Cache the HTML itself. Marketing and docs pages rarely change per visitor; serve them from a CDN or static cache so no application code runs per request.
2. If you are on serverless hosting, cold starts are a common cause: keep the function warm or pre-render the page.
3. Move slow work (database queries, third-party API calls) out of the first response; render the shell fast and fill personalised details after.

• Run: curl -o /dev/null -s -w "%{time_starttransfer}\n" https://yoursite.com and watch the number drop.
• Hit Re-scan above: "Time to first byte" improves a bucket at a time (under 500ms passes).

• Don’t answer fast with an empty shell that loads content by JavaScript. That trades this warning for the much worse "content requires JavaScript" failure, since most agents never run your scripts.
• Don’t special-case known crawler user agents onto a faster path. Serving different performance to different audiences is the pattern crawlers penalise when they detect it.

MCP (Model Context Protocol) is how AI assistants call tools. A server card is a small JSON file at /.well-known/mcp/server-card.json that announces "this site offers these tools" so an assistant can find them without scraping your pages. We requested that file from your site and found none.

Assistants that support MCP look for the card at that exact address. With one, "book an appointment on yoursite.com" can become a tool call instead of screen-driving your UI. This is an emerging standard, so it weighs little today; publishing early costs one static file and makes you visible to the assistants adopting it.

1. If you already run an MCP server, describe it: name, description, and endpoint URL in the card.
2. If you don’t, a minimal truthful card naming your site and linking your API docs is still valid. The generated starter in the Fix it section below is filled in for this site.
3. Serve it at exactly /.well-known/mcp/server-card.json with content type application/json.

• Open yoursite.com/.well-known/mcp/server-card.json and confirm it returns your JSON, not a 404 page.
• Hit Re-scan above: "MCP server card" flips to PASS.

• Don’t advertise tools that don’t exist. An assistant that calls a declared endpoint and gets a 404 learns to skip your card entirely.
• Don’t let your framework rewrite /.well-known/* to your single-page app; the file must return raw JSON, not your HTML shell.

RFC 8414 metadata is a JSON file at /.well-known/oauth-authorization-server that tells software where your sign-in endpoints live: where to send a user to authorise, where to exchange tokens. We requested it and found none. This only applies if your site has accounts or an API that third parties connect to.

Agents acting on a user’s behalf (booking, purchasing, managing an account) need permission first, and OAuth is how they get it without ever seeing a password. The metadata file is how they discover your endpoints programmatically instead of a developer hand-configuring each integration. No file means every agent integration starts with manual setup, so most never start.

1. If you run an OAuth authorization server (Auth0, Keycloak, Okta, or your own), enable its metadata endpoint; most providers publish RFC 8414 metadata with a setting.
2. Serve or proxy it at /.well-known/oauth-authorization-server on this domain.
3. If your site has no user accounts or API, this check simply isn’t your priority; it is one reason it carries a small weight.

• Open yoursite.com/.well-known/oauth-authorization-server and confirm JSON with an "issuer" field matching your domain.
• Hit Re-scan above: "OAuth authorization metadata" flips to PASS.

• Don’t publish metadata naming endpoints that don’t answer; broken discovery is worse than none, because clients trust the file over their own guesses.
• Don’t copy another provider’s metadata as a template and leave their URLs in it. Every URL in the file must be yours.