The robots.txt file was designed in 1994 for one kind of visitor: the search crawler. In 2023 the visitor list exploded. OpenAI announced GPTBot in August 2023, Google added the Google-Extended control in September 2023, and every answer engine since has shipped its own user agent. Site owners now manage three distinct bot classes with one blunt file, and the most common configuration mistakes block the wrong class entirely.
Three bot classes, three different bargains
Training crawlers such as GPTBot and CCBot collect text for model training; blocking them is a pure policy choice with no effect on your visibility today. Search-class agents — OAI-SearchBot, PerplexityBot and their peers — index pages so answer engines can cite them; blocking these removes you from AI answers the same way blocking Googlebot removes you from search. User-triggered fetchers act on a single human request for a single URL, and vendors document them as a separate class precisely because a human is behind each fetch.
Write the bargain you actually want. A publisher can refuse training and still welcome search: disallow GPTBot and CCBot, allow OAI-SearchBot and PerplexityBot. The two decisions are independent, and conflating them is the single most expensive robots.txt mistake of the last three years.
The Google-Extended misunderstanding
Google-Extended is not a crawler. No request ever arrives with that user agent; it is a control token that Googlebot reads in robots.txt to decide whether your pages feed Gemini training. Google states this directly in its crawler documentation. Blocking Google-Extended changes nothing about crawling, indexing or ranking — and allowing it grants nothing to search. Sites that "block Google AI" by disallowing Google-Extended while leaving Googlebot untouched have made a training-policy statement, not an access decision.
The 5xx trap and other silent failures
RFC 9309 section 2.3.1 rules that a robots.txt returning a server error means the crawler must assume the entire site is disallowed. A misconfigured CDN that serves 500 on /robots.txt therefore delists you from every compliant bot at once — search and AI alike — while the page itself works fine in a browser. The same RFC obliges parsers to accept at least 500 KiB of robots.txt, sets longest-match precedence for path rules, and defines that an empty Disallow line means everything is allowed.
Audit the file like code, because it is code. Fetch it from outside your network and confirm the status is 200 or 404. Check that path rules match your real URL structure — a Disallow: /report does not cover /en/report, because matching is prefix-based on the literal path. Test the specific user agents you care about against the specific paths that earn citations.
A checklist that survives contact with production
Keep four invariants and the file stays honest: robots.txt answers in under a second with 200 or 404; every agent class you name has an explicit, intended rule; training policy and search policy are decided separately; and every rule change is tested against the three or four URLs that actually earn citations. Sites that hold these four lines survive vendor list churn without emergency edits — new bot names slot into an existing class instead of forcing a rethink.
Sources
The RFC is the binding text; the three vendor pages define the user agents and their classes. Vendor bot lists change several times a year, so bookmark the pages themselves rather than summaries of them.