A chatbot grounded in your own documentation is a retrieval system with a conversational front end. It fetches your pages, cuts them into passages, turns each passage into a vector and hands the closest few to a language model. All 4 of those steps happen before the model writes a word, and each one fails quietly. Teams see the wrong answer, blame the model and swap it for a larger one. The larger model repeats the wrong answer, because the passage it was handed was already wrong.
A grounded chatbot starts with the same HTTP GET as a crawler
Retrieval-augmented generation was described by Lewis and colleagues in a 2020 paper, and its core move still describes most site chatbots: retrieve first, generate second. The retrieval half is ordinary web plumbing — something requests a URL, strips the markup, splits the text, embeds each piece and stores it. Your ingestion script and GPTBot run that identical first step against the identical HTML.
The shared first step is why these look like 2 problems and are 1. A page an answer engine cannot read is a page your own assistant cannot read. The difference is visibility: a customer opens a ticket when your chatbot is wrong, and nobody reports the citation you never received.
A JavaScript shell hands the ingester an empty document
Client-rendered documentation is the quietest failure in this pipeline. The server returns a near-empty container, the browser fills it, and any fetcher that skips JavaScript stores nothing. Google splits the work into 3 phases — crawl, render, index — in its JavaScript SEO guide, and queues the render pass separately. Ingestion scripts and the major AI fetchers take a single pass instead: 1 GET, no render, no hydration, no client router.
Measure it instead of assuming. Fetch the page with curl, strip the tags, count the sentences that survive. Lekta reports the gap as a JS dependency coefficient between 0 and 1 — the share of the main text that exists only after rendering. At 0.01 and under the difference is measurement noise and the page keeps full credit; at 0.35 the check is already down to a warning; at 0.5 more than half the visible text is invisible to a fetcher that never renders. A chatbot built on that corpus is not hallucinating. It is answering from an empty shelf.
Chunk boundaries decide what the bot is able to say
Chunk size, overlap and the number of passages retrieved are per-tool settings, not a standard: documented defaults run from a few hundred tokens to several thousand characters, and the retrieved count from 2 to 50. Write for the chunk covers the mechanics. Whatever the setting, each retrieved passage arrives alone: context that sat 2 paragraphs above it does not travel with it.
Prose written for continuous reading breaks here. A section opening with “This approach also works when…” loses its referent the moment the chunk is lifted, and the model fills the gap by guessing. Sections that name their subject in the first sentence survive the cut intact. Writing them that way costs nothing and returns more than any retrieval parameter you can tune afterwards.
Embeddings compare meaning, not the headings you wrote
Retrieval matches a question vector against passage vectors. Lekta embeds with multilingual-e5-small, in an int8 build — a 384-dimension model covering 100 languages. A small multilingual embedder like it is a common choice behind site chatbots. A heading earns its retrieval weight only when the body beneath it discusses that heading; a decorative heading contributes nothing to the vector and costs you the match.
Two sections restating 1 idea produce 2 nearly identical vectors that compete for the same retrieval slots, splitting relevance a single strong section would hold. Merge them, keep the better example from each, and point the freed heading at a question your corpus does not answer yet.
An undated page turns a stale price into a confident answer
A retrieved passage carries no memory of when it was written. If your pricing changed in March and the chunk store still holds the January text, the model states the January figure with complete confidence, because stating the context it was given is exactly what it does.
A machine-readable date is the cheapest defence available. An Article object carrying datePublished and dateModified gives the ingester a field to sort and expire on, and hands an answer engine the same freshness signal. Print the date where a reader sees it too: a date living only in JSON-LD serves the parser and leaves the human guessing.
Audit the corpus before replacing the model
The reflex when a grounded chatbot answers badly is to change models. That is the expensive move and the wrong one. Take the failing question, look at which passages were retrieved, and read them the way the model received them — alone, stripped of everything around them. The passage turns out to be empty, stale or missing its subject far more often than a capable model reasons badly over a good passage.
The corpus fix and the answer-engine fix are one piece of work. Server-rendered HTML, self-sufficient sections, honest dates and concrete claims raise the accuracy of your own assistant and your odds of being quoted elsewhere, out of a single editing pass. Audit the pages first; the model is rarely the bottleneck.
Sources
The arXiv paper is the one that named retrieval-augmented generation; the model card documents the embedder’s 384 dimensions and 100-language coverage; the Google guide covers rendering and the schema.org definition covers dates. All 4 links point at primary, maintained documents.