The Vibe Coding Index
How agent-ready are the sites people build with AI app builders? This page is the method — the sampling frame, the filters, the seeded draw, and the conditions under which we would refuse to publish — written down before any site was scanned.
No site has been scanned. Nothing here says anything about any AI builder, and any number you see below is a threshold we committed to, never a measurement. When an edition runs, the result appears here — including if the gates below refuse it, which is reported as an outcome rather than quietly retried.
The sample is sites on a builder’s default subdomain — not “sites built with that builder.” The distinction is the study’s biggest limitation and it is deliberate: a project that graduates to a custom domain is systematically more finished than one that never left the default host, so this measures the part of each platform’s output that stayed put. Reported that way, always.
Frozen so the set cannot quietly grow to include whichever platform makes a result more interesting. Adding one is a new edition, not an edit. The controls are not padding — if AI-built sites score badly, the reader’s real question is whether they score worse than ordinary small sites on a generic host.
Order matters twice. Between rules it makes the removal counts interpretable — each is “removed by this rule, having passed the earlier ones” — and relative to the draw it matters more: filtering a random sample after seeing the results silently redefines the population as “sites that survived a filter we chose once we knew what it would remove.” The count removed by each rule is published with any result.
| Q1 | Resolves and responds over HTTPS | A name in Certificate Transparency with nothing behind it is not a site. |
| Q2 | Final status is 2xx | A 404 or a parked page is not a published site. |
| Q3 | Not the platform's unmodified starter template | Measuring the template measures the platform's default, not what anyone built. Detected by exact match against a captured fingerprint per platform. |
| Q4 | At least 3 distinct content elements and 200 characters of body text | Below that it is a placeholder. The thresholds are frozen here rather than tuned once the results are visible. |
| Q5 | robots.txt does not disallow our user agent | A site that declines to be scanned leaves the denominator entirely — it is excluded, never scanned anyway and never counted as a failure. |
From the qualified set, 200 per platform, drawn with a seeded PRNG whose seed comes from the edition key — so the draw is deterministic, published, and reproducible by anyone holding the frame manifest. If a platform has fewer than 200 qualified sites we take all of them and report the true n. Never topped up from a different platform, a different capture date, or a second query: a sample assembled to reach a round number is not a sample.
Numbered in evaluation order, so the number is the order. None is advisory and none takes an override argument — a gate you can pass a flag to is a suggestion, and the moment you want to override one is exactly the moment it is working.
| V1 | One instrument | Void the edition if scans span more than one rubric version or measurement profile. |
| V2 | The population did not move | Void if the frame manifest hash no longer matches the capture. |
| V3 | Scan completion | Drop a platform if fewer than 80% of drawn sites produced a score. |
| V4 | Qualification transparency | Block publication unless the count removed by each of Q1–Q5 is recorded. |
| V5 | Platform size | Drop a platform with fewer than 100 scored sites rather than publish it thin. |
| V6 | A control survived | Without one, "AI builders are bad at this" cannot be separated from "small sites are bad at this" — stated in the headline, not the appendix. |
| V7 | Spread, not a mean | A mean-only row cannot be constructed: the distribution is the only publishable form. |
A rehearsal of the pipeline on 30 randomly drawn lovable.app candidates qualified none of them: 24 returned HTTP 404, two were the unmodified starter template, and four were placeholders. That is not a bug — it is the finding that a Certificate Transparency frame is a census of names, and most names never become sites.
So the honest position today is that the frame is not yet wide enough to reach the 100 scored sites V5 requires for any platform. That is a problem to solve by capturing more of the population, and explicitly not by loosening Q1–Q5 until enough sites survive — which would be choosing the filter after seeing what it removes, the exact failure the ordering above exists to prevent.
- Distributions, never a single mean — decile boundaries are the publishable form.
- Never individual site names. This measures platforms, and naming somebody’s side project as an example of bad work is not a finding.
- A refused edition is reported as refused, with the gate that stopped it.
- Nothing here claims agent-readiness causes anything. That question belongs to the citation study, which is registered and has no data.
Full pre-registration, including the frame capture procedure and the manifest format: docs/research/VIBE-CODING-INDEX.md in the repository. Other measurements and registered methods are listed at /research.