Every agent-readiness tool gives you a number. None of them tells you how often that number is wrong. We publish ours, and this is what it says today: on a frozen corpus of hand-labelled cases, 127 observations produced 1 false positive and 0 false negatives, for 99.2% agreement with the labels.
That sentence is true and it is also the most misleading thing on this page, so here is the part most vendors would leave out.
The corpus is a stress test, not a sample
Those cases were written to defeat our own rules. Pages that look compliant and are not. Markup that validates and lies. Metadata stuffed with keywords that passes a presence check. Performance on a corpus like that is a lower bound on how the rules behave when somebody is trying to fool them — it is not an estimate of how often a score is right about a random website, and it must never be quoted as one.
If you take one thing from this article: “99.2% agreement on an adversarial corpus” and “99.2% accurate” are different claims, and only the first one is ours.
Why we publish no per-check rate at all
A precision figure needs a denominator that can carry it. We set that floor at 3 labelled cases. Right now the largest denominator any single check has is 2, so the number of per-check rates we can publish is 0.
A check with one true positive and no false positives has a precision of “100%” in exactly the way a coin that landed heads once is “100% heads”. Publishing that would be the same unearned precision we criticise when a tool scores a site out of a hundred and shows no working. So the counts are published and the percentages are withheld, with the reason stated per row.
The check we get wrong, and why it was written down first
One check disagrees with its labels: the page-title check reports a keyword-stuffed title as fine. That is a real false positive, and it was recorded in the corpus as a known limitation before it showed up as an error — because the check measures presence and length, and detecting low-quality-but-present metadata needs a judgement a deterministic rule cannot make.
Writing a limitation down before it embarrasses you is the whole discipline. It also means nobody can later claim these checks measure quality, and any proposal to raise their weight has to answer that case first.
What we cannot measure at all
Coverage is the part everyone omits. A check with no labelled case is not a passing check — it is a check we have no evidence about, and a table that quietly left those rows out would look more complete than it is. The uncovered checks are listed by name, with the reason each one cannot be calibrated offline.
Why any of this matters to you
Because a score you cannot audit is a marketing number. If a tool tells you that you scored 62, the useful questions are: how often is that check wrong, on what evidence, and what does it not look at? If those answers do not exist publicly, the score is an opinion with a typeface.
Ours are at /rubric/accuracy, regenerated from the corpus and committed, so every change to them arrives as a reviewed diff rather than at the next deploy.