How often the score is wrong
Readiness scores in this category are published without an error rate. Ours has one, measured against a frozen corpus of hand-labelled cases, and this page is it — including the parts that do not flatter us: where we disagree with ground truth, which rates the evidence is still too thin to state, and which checks no ground truth covers at all.
These cases are adversarial by construction: each one was written to defeat a specific rule. The results are a stress test, not a sample of the web, so they describe how the rules behave when someone is trying to fool them and not how often a score is right about a random site.
126 of 127 observations agreed with the label (99.2%). Corpus v1 · rubric 2026.11.0. A check that returned SKIP on a case it could have answered counts as an observation and never as an agreement: a rule must not be able to improve its rate by staying silent.
No per-check precision or recall figure on this page is publishable yet. A rate needs a denominator that can carry it. We set that floor at 3 labelled cases, and the largest denominator any single check currently has is 2. Below the floor a percentage is one or two observations wearing a percentage sign — the same unearned precision we criticise when a vendor scores a site out of a hundred and shows no working.
So the corpus is doing a narrower job than a percentage would imply. It is a regression harness with an activation gate: it proves a check distinguishes the cases it has been shown, and it fails the build when a rule starts disagreeing with a label it used to match. It is not yet a basis for “this check is 94% precise”, and this page will not say so until the cases exist. Growing those denominators is the work; claiming them early would be the shortcut.
markup.title (Page title) is the only check that disagrees with its labels: 1 false positive and 0 false negatives across 8 labelled cases. The failing case is adv-002-keyword-stuffed-metadata.
adv-002-keyword-stuffed-metadata — ACKNOWLEDGED GAP: the check measures presence and length, so keyword-stuffed metadata passes. Detecting low-quality-but-present metadata needs a judgement the deterministic class cannot make. Recorded here so nobody claims these checks measure quality, and so any future proposal to raise their weight has to answer this case first.
This is near the top of the page rather than the bottom because a methodology page that opens with its cleanest rows is a brochure.
Ordered worst first. TP is a defect the check caught, FN one it missed, FP a defect it reported that was not there, TN a clean case it correctly left alone. Rates appear only where the denominator reaches 3 labelled cases; everywhere else the counts are the honest answer, and they are shown rather than hidden.
| Check | TP | FP | FN | TN | Skipped | Precision | Recall |
|---|---|---|---|---|---|---|---|
| markup.title2 flagged · 1 defective | 1 | 1 | 0 | 6 | 0 | — | — |
| actionability.agent_skills2 flagged · 2 defective | 2 | 0 | 0 | 3 | 0 | — | — |
| actionability.commerce_protocols1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| actionability.web_mcp1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| auth.cookie_consent_blocks_content1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| auth.paywall_or_login_wall_on_landing1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| coherence.jsonld_name_matches_visible1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| coherence.jsonld_price_matches_visible1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| coherence.language_declared_matches_content1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| content.headings_hierarchy1 flagged · 1 defective | 1 | 0 | 0 | 4 | 0 | — | — |
| content.js_required_for_primary_content1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| content.text_ratio1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| discoverability.api_catalog2 flagged · 2 defective | 2 | 0 | 0 | 2 | 0 | — | — |
| discoverability.link_headers1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| discoverability.mcp_server_card2 flagged · 2 defective | 2 | 0 | 0 | 3 | 0 | — | — |
| discoverability.oauth_authorization_server2 flagged · 2 defective | 2 | 0 | 0 | 3 | 0 | — | — |
| discoverability.oauth_protected_resource1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| discoverability.web_bot_auth1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| markup.canonical2 flagged · 2 defective | 2 | 0 | 0 | 4 | 0 | — | — |
| markup.language1 flagged · 1 defective | 1 | 0 | 0 | 5 | 0 | — | — |
| markup.meta_description2 flagged · 2 defective | 2 | 0 | 0 | 6 | 0 | — | — |
| operability.filters_addressable1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| operability.primary_action_in_dom1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| performance.full_render_ms1 flagged · 1 defective | 1 | 0 | 0 | 3 | 0 | — | — |
| performance.payload_kb1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| performance.ttfb_ms1 flagged · 1 defective | 1 | 0 | 0 | 3 | 0 | — | — |
| reachability.status1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| readability.markdown_negotiation1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| robots_txt.allows_known_agents1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| robots_txt.content_signals1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| robots_txt.present1 flagged · 1 defective | 1 | 0 | 0 | 4 | 0 | — | — |
| sitemap.present1 flagged · 1 defective | 1 | 0 | 0 | 2 | 0 | — | — |
| structured_data.jsonld_present2 flagged · 2 defective | 2 | 0 | 0 | 6 | 0 | — | — |
| structured_data.type_coverage1 flagged · 1 defective | 1 | 0 | 0 | 1 | 0 | — | — |
| structured_data.validates1 flagged · 1 defective | 1 | 0 | 0 | 3 | 0 | — | — |
The corpus asserts nothing about 1 scored check. It still contributes to a score; we simply have no measured error rate for it, and a table that omitted it would look more complete than it is.
- journey.primary_probellm_assisted: the runner returns SKIP unless ANTHROPIC_API_KEY is set, so an offline fixture can only observe the disabled path. Calibrating it requires the recorded-provider harness in packages/ai, tracked separately from this corpus.
A check must pass an activation gate before it is allowed to contribute to a score, and the gate requires labelled cases on both sides: a rule that has never been shown a page it should reject has not been shown to distinguish anything. These numbers are regenerated from the corpus and committed, so any change to them appears in a reviewed diff rather than at the next deploy. The rubric itself, with weights and grade bands, is at /rubric.