Why not just use the certificate log and filter later?

Because filtering after you have seen the results lets the filter be chosen to produce them. The qualification rules have to be fixed before the sample is drawn, otherwise the population silently becomes "whatever survived a rule we picked once we knew the answer".

How many of the names were real sites?

In the first run, none of 30. Twenty-four returned HTTP 404, two were the platform default template, and four were placeholders with almost no content.

Does this mean the platform has no real sites on it?

No. It means a certificate log cannot tell you which ones they are, and a sample drawn from one without a qualification step is mostly abandoned deployments. Finding the real sites needs a wider capture and the filters applied first.

A certificate log is not a list of websites

We drew 30 sites from a certificate log. Twenty-four were 404s. Here is what that means for anyone counting the web.

Certificate Transparency is a public, append-only log of every TLS certificate issued on the internet. Every time somebody deploys a site on a platform like Lovable, Replit or Vercel, a certificate gets issued for their subdomain and lands in that log. It looks like the perfect sampling frame: a near-complete census of everything that exists, free to query, updated continuously.

We built a study on it. Then we ran the pipeline for the first time against 30 randomly drawn lovable.app candidates, and none of them qualified.

What the 30 actually were

Twenty-four returned HTTP 404. Two were the platform’s unmodified starter template. Four had less real text than a business card. Zero were a site somebody had built and shipped.

That is not a bug in the pipeline. It is the finding: a certificate log is a census of names, not of sites. A certificate is issued when somebody clicks “deploy”. Nothing revokes it when they wander off ten minutes later, and nothing marks it as abandoned. The log faithfully records an intention that mostly did not survive the afternoon.

Why this matters beyond our study

Any claim of the form “there are N sites built with X” that is derived from certificates or DNS is counting names. If you have seen a chart of platform adoption assembled this way — and they circulate constantly — the denominator is doing something very different from what the headline implies.

The honest version needs a qualification step between the census and the sample: does the name resolve, does it answer, is it the default template, does it have content. Each of those removes a large fraction, and the fraction they remove is itself the interesting number.

The mistake we did not make, and it was available

When a filter removes 100% of a sample, the tempting fix is to loosen the filter until enough sites survive. It would have worked. We would have had data by the afternoon.

It would also have been choosing the filter after seeing what it removes, which quietly redefines the population as “sites that survived a rule we picked once we knew the answer”. The qualification rules were fixed in a public pre-registration before any site was probed, precisely so that this afternoon’s inconvenience could not rewrite them.

So the study does not have results yet, and the reason is published rather than hidden. The method, the rules, the gates and this run record are at /research/vibe-coding-index.

If you are sampling the web yourself

Three things worth stealing. Capture your frame once and commit it, because the source moves under you — two captures of the same query hours apart differed by four sites. Write the qualification rules down before you look at what they remove. And publish the removal counts with any result: a filter that quietly ate 90% of the frame changes what the study is about, and only the counts make that visible.