Search engines rank pages; answer engines quote them. When ChatGPT, Perplexity or an AI browser agent answers a question, it fetches a handful of pages, cuts them into passages and cites the passages that survive the trip. A page that renders beautifully for humans loses at this game in four separate places, and none of them show up in a normal analytics dashboard.
Access: the fetch that never happens
Every citation starts with a plain HTTP GET, and robots.txt decides whether that GET is welcome. The file format dates back to 1994 and was only standardised in September 2022 as RFC 9309. Three failure modes dominate in practice: a robots.txt that returns a 5xx error (RFC 9309 tells crawlers to treat the whole site as disallowed), a blanket rule written years ago that now blocks retrieval bots along with scrapers, and firewall rules that serve challenge pages to any non-browser user agent.
The fix costs one afternoon: serve robots.txt with a 200 or a 404, never a 500; list the agent classes you accept explicitly; and test the page with a plain GET from outside your network. A blocked fetch is worth 0 citations no matter how good the content is.
Indexability: server HTML versus a JavaScript shell
Retrieval systems work with the HTML the server sends, and only some of them run JavaScript afterwards. Google documents 3 separate phases — crawling, rendering and indexing — in its JavaScript SEO guide, and queues the render pass on its own schedule. Most answer-engine fetchers skip that pass entirely: no render, no hydration, no client router. A single-page app that ships an empty <div id="root"> ships exactly nothing to those systems.
Measure the gap directly: fetch your page with curl, strip the tags and count the sentences that remain. Lekta grades this indexability layer with a JS dependency coefficient between 0 and 1 — the share of visible text that exists only after rendering. Pages under 0.01 are effectively static; pages above 0.5 gamble their entire content on a render pass that most fetchers never run.
Answerability: content shaped for quotation
A fetchable, parseable page still competes on whether its passages answer questions. Concrete data wins: dates, numbers, names and units give a language model something to quote and a fact-checker something to verify. Structured markup helps the same way — an Article object with datePublished and dateModified tells the engine how fresh the page is, and an og:type declaration tells it what kind of document it is reading.
Structured markup is where that shape becomes machine-readable. Declare the document type once and let the vocabulary carry the rest: an Article object states what the page is, an og:type declaration states how it should be previewed, and both travel with the passage when an engine lifts it.
Recency: a date the machine and the reader both see
Recency is the strongest in-page citation signal measured so far, and it is the one most often half-implemented. A dateModified stamp in JSON-LD with no date anywhere on the rendered page reads as decoration, and several engines discount it. Print the date where a reader sees it, keep the markup in sync, and update both when the content actually changes.
Recency does not mean churn. Publishing a fresh timestamp on unchanged text is exactly the behaviour engines learn to distrust, so Lekta reports the age of a page but never scores it — what the recency layer grades is whether a date exists at all and whether a reader can see it.
Where to start on a real site
Audit one representative article page, not the homepage. Landing pages are navigation; articles are what answer engines quote. Check the four layers in order — a robots.txt problem invalidates everything downstream, so fixing layers out of order wastes work. The whole loop for one page takes under an hour, and the access and indexability fixes usually carry over to the entire site at once.
Sources
All four references are primary: the RFC text itself, the vocabulary definitions at schema.org and ogp.me, and Google’s own crawler documentation. Read them in that order and they map onto the four layers above.