Engine versions and what changed

Agentmap and Host lines no longer split a robots.txt group

Our robots.txt reader now handles Agentmap and Host lines the way RFC 9309 and Google do. The agent list grew from 36 to 38 tokens with Google-Agent and Google-GeminiNotebook, two user-triggered agents; like the rest of that class they are reported for information and never change a score. A set of corrections that change no score shipped with it; most make report texts claim only what their sources show.

• An Agentmap: or Host: line placed between two User-agent lines used to cut the first agent off from the rules that follow, so we read that agent as allowed. We now read both agents as one group, as RFC 9309 and Google do. If the first agent is an AI search bot and those rules block it, your report now shows the block. Crawl-delay and Content-signal lines still end a group, on purpose: most sites that give one bot only a Crawl-delay mean "slow down", not "block", and reading them the other way would show bingbot blocked on sites that never blocked it.

Measured — Measured before release: zero movement across all six targets in our frozen reference set, and zero changed allow/block decisions when the new reader was compared with the old one on two samples of real robots.txt files (177 and 89 sites). Pages that can move are the ones that put an Agentmap: or Host: line between two User-agent lines.

The AI agent list doubled: 36 tokens instead of 18

We re-read every vendor's own crawler documentation and rebuilt the list your robots.txt is checked against. It grew from 18 tokens to 36, and the search-class group — the one that decides whether an answer engine can index you at all — grew from 6 to 15. Nine internal corrections shipped with it, none of which change a score.

• Meta AI now has a search indexer in the list (Meta-WebIndexer). If your robots.txt blocks Meta tokens as a group, that block now costs you a measured answer-engine surface it did not cost before.
• Amazon separated training from search. Blocking Amazonbot to refuse training no longer implies losing Amazon-side visibility — that is governed by Amzn-SearchBot, a separate token. Pages penalised for this before can score higher now.
• Mistral, Huawei, Moonshot (Kimi), DuckDuckGo, Yandex, Kagi and Ai2 entered the list. Mistral and Yandex publish a training-only opt-out, so refusing training there no longer costs you citations.
• Blocking every well-known AI search bot is no longer automatically the strictest failure. If you left Yandex, Huawei or Kimi unnamed, you are genuinely still visible there, and the report now says so instead of capping you.
• Applebot, Kagibot and Amzn-SearchBot fall back to your Googlebot rules when you do not name them. Writing Disallow for Googlebot quietly blocks all three; the report now flags the relationship.

Measured — Measured before release: zero movement across all six targets in our frozen reference set. Coverage doubled, scores did not move. Pages that will move are the ones blocking Meta tokens as a group (down) and the ones blocking Amazonbot to refuse training (up).

Second wave of the independent scoring audit

Sixteen findings from the 2026-08-20 independent audit closed in one release. Two of them can move a page grade; the other fourteen are internal corrections with no visible effect on scores.

• Recency: the date you show readers must match a date you declare to machines. A date-shaped string anywhere on the page used to be enough.
• A+ now requires an unrounded full score. Pages that rounded up from 99.5 to 100 were getting A+; your points are unchanged, only A+ eligibility tightened — so a report can show 100 points with an A.

Measured — Measured before release: in our frozen reference set one target of six moved (Wikipedia, 100 → 94, because it declares 2001 to machines and shows 2026 to readers); the broken-page corpus did not move at all.

First wave of the independent scoring audit

An independent review of the scoring engine produced 23 confirmed findings. This release closed the high-severity half.

• Pages that read like an article but carry no article markup are now graded as articles. Deleting markup no longer raised a grade.
• The robots.txt and X-Robots-Tag readers were corrected, and robots rules are now evaluated against the page a redirect actually lands on.
• A crashed rule can no longer inflate a score silently — the report is marked degraded instead.

Page-type detection: dominance instead of counting

A page with several <article> elements was blindly treated as a listing, which under-measured real articles that happen to embed related cards.

• If the largest article carries at least 60% of the page text (and at least 500 characters), the page is measured as an article rather than a collection.

Four layers instead of three

The score was restructured so that each layer answers one question a reader can check.

• Layers became Access 25 · Indexability 25 · Answerability 30 · Recency 20. Recency became a layer of its own instead of a check inside another layer.

Semantic section signals

Two scored checks were added using a frozen multilingual embedding model, pinned to a fixed revision and run on CPU so the same page always yields the same number.

• Answerability now measures whether a section actually answers its own heading, and whether sections stand on their own when quoted out of context.

A+ means a full score

The grade definitions were tightened to match the published plan.

• A+ now requires a full score and zero warnings. A page scoring 97–99 without warnings is an A.
• Warnings do not change the number; their only effect is on the grade.

First prototype

The first working engine: an AI-crawler matrix over robots.txt, a JavaScript dependency measure, content freshness, a severity-weighted score and the partial-coverage cap.

No change that moves an existing grade.