How the rules work
Each crawler follows the group in robots.txt that names it. If none does, it follows the group for all crawlers, User-agent: *. That's why a group for each AI crawler is the clearest way to say what you want: it wins over your general rules.
| Crawler kind | If you block it |
|---|---|
| Search crawlers | That AI can't find or cite your pages in its answers |
| Visits for a user | That AI may not open your page when someone asks about it |
| Training crawlers | Your pages aren't used to train that company's future models. Search answers aren't affected |
After you upload it
- 1
Check it
Run the AI crawler checker to make sure every rule reads the way you meant.
- 2
Check your firewall
A firewall, CDN or security plugin can block a crawler that robots.txt allows.
- 3
Give it a day
OpenAI and Perplexity say changes take about 24 hours to take effect.
Questions
Will this replace my whole robots.txt?
Only if you want it to. If you load your current file, we keep every rule that isn't about these AI crawlers and your Sitemap lines, and put the AI crawler rules on top. Read it through before you upload it.
Which AI crawlers should I allow?
Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot) if you want to be found and cited in AI answers. Training crawlers are your choice and don't affect search answers.
Does Perplexity-User follow robots.txt?
Perplexity says Perplexity-User, which fetches a page because a user asked about it, generally ignores robots.txt. It's in the list so your file states your intent.
My rules for all crawlers block everything. Does that block the AI crawlers?
Not the ones listed here. A crawler follows the group that names it, so a User-agent group for OAI-SearchBot with Allow: / wins over User-agent: * with Disallow: /.