robots.txt in the AI era: search bots, training bots and user-triggered fetchers
AI crawlers split into three classes with different stakes. Blocking the wrong one costs citations without protecting anything. How to write robots.txt rules that say what you mean.
3 min · RFC 9309