Keeping citation and training permissions separate is the point: it is how brands avoid accidentally opting out of being cited while trying to opt out of being trained on. Toggle on = Allow, off = Disallow.
#
Directives follow each vendor's published user-agent tokens. robots.txt is honoured by convention, not enforced, verify with your logs.
Why two lists
Search/citation crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are how AI answers cite you, you almost always want them allowed. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) harvest data for model training, a separate decision. Conflating them is how companies accidentally disappear from citations while trying to opt out of training.
After deploying, confirm nothing else (a WAF, a CDN rule) is silently blocking the bots you just allowed.
How to use it
- Toggle allow or disallow per AI crawler.
- Separate citation bots from training bots.
- Copy the generated robots.txt to your site root.
Frequently asked questions
Can I allow AI citation bots but block training bots?
Yes, and you should decide deliberately. Citation crawlers (OAI-SearchBot, PerplexityBot, Google-Extended for AI surfaces) are the ones that quote you; training crawlers (GPTBot, CCBot) feed model training. This generator lets you allow one class and control the other.
Will blocking GPTBot stop me being cited?
It can. Some crawlers serve both training and retrieval, and over-blocking removes you from the pool that quotes you. The generator flags which bots affect citations so you do not block the ones you want.
Is the robots.txt for AI Generator free to use?
Yes. The robots.txt for AI Generator is completely free, with no sign-up, no usage limits and no watermark on the output.
Is my data private?
Yes. The robots.txt for AI Generator runs entirely in your browser. Nothing you paste or enter is uploaded, stored or sent to any server.
Related tools and reading
Keep going with the playbooks and tools behind this one.