The robots.txt file was designed in 1994 for one kind of visitor: the search crawler. In 2023 the visitor list exploded. OpenAI announced GPTBot in August 2023, Google added the Google-Extended control in September 2023, and every answer engine since has shipped its own user agent. Site owners now manage three distinct bot classes with one blunt file, and the most common configuration mistakes block the wrong class entirely.
Three bot classes, three different bargains
Training crawlers such as GPTBot and CCBot collect text for model training; blocking those two is a pure policy choice with no effect on your visibility today. Search-class agents — OAI-SearchBot, PerplexityBot and their peers — index pages so answer engines can cite them; blocking these removes you from AI answers the same way blocking Googlebot removes you from search. User-triggered fetchers act on a single human request for a single URL, and vendors document them as a separate class precisely because a human is behind each fetch.
Write the bargain you actually want. A publisher can refuse training and still welcome search: disallow GPTBot and CCBot, allow OAI-SearchBot and PerplexityBot. The two decisions are independent, and conflating them is the single most expensive robots.txt mistake of the last three years.
The Google-Extended misunderstanding
Google-Extended is not a crawler. No request ever arrives with that user agent; Google crawls with its existing user agents and reads the token in robots.txt as a control. Google’s crawler documentation gives it two jobs: whether your pages train future Gemini models, and whether they ground answers in Gemini Apps and Vertex AI. So blocking it has a real cost — Gemini will rarely, if ever, cite you — but by Google’s account it does not change inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search: Google ties them to Googlebot and to preview controls such as nosnippet, and since 31 August 2026 to a Search generative AI control in Search Console that no outside audit, Lekta included, can see. Google’s help page for that control adds a third job for Google-Extended: it also limits training of the models behind Search’s generative AI features.
One observational study by Grossman et al. (SIGIR 2026; US results from December 2025) did see sites that block Google-Extended cited less in AI Overviews than in their organic results; the authors call it surprising and their analysis descriptive, so read it as an open question, not a measured cost. Disallowing Google-Extended while leaving Googlebot untouched is a decision about training and Gemini, not an opt-out from AI Overviews or AI Mode.
The 5xx trap and other silent failures
RFC 9309 section 2.3.1 rules that a robots.txt returning a server error means the crawler must assume the entire site is disallowed. A misconfigured CDN that serves 500 on /robots.txt therefore delists you from every compliant bot at once — search and AI alike — while the page itself works fine in a browser. The same RFC obliges parsers to accept at least 500 KiB of robots.txt, sets longest-match precedence for path rules, and defines that an empty Disallow line means everything is allowed.
Audit the file like code, because it is code. Fetch it from outside your network and confirm the status is 200 or 404. Check that path rules match your real URL structure — a Disallow: /report does not cover /en/report, because matching is prefix-based on the literal path. Test the specific user agents you care about against the specific paths that earn citations.
A checklist that survives contact with production
Keep four invariants and the file stays honest: robots.txt answers in under a second with 200 or 404; every agent class you name has an explicit, intended rule; training policy and search policy are decided separately; and every rule change is tested against the three or four URLs that actually earn citations. Sites that hold these four lines survive vendor list churn without emergency edits — new bot names slot into an existing class instead of forcing a rethink.
Sources
The RFC is the binding text; the vendor pages define the user agents and their classes, and the SIGIR 2026 paper is one observational snapshot. Vendor bot lists change several times a year, so bookmark the pages themselves rather than summaries of them.
- RFC 9309 — Robots Exclusion Protocol
- OpenAI — bot and crawler documentation
- Google Search Central — Google crawlers and fetchers
- Google — Google-Extended (common crawlers)
- Google Search Console Help — Search generative AI control
- Google Search Central — Robots meta tag specifications (nosnippet)
- Perplexity — crawler documentation
- Grossman et al. — How Generative AI Disrupts Search (SIGIR 2026, arXiv)