1. Home
  2. Free tools
  3. robots.txt generator

Free tool

robots.txt generator for AI crawlers

Choose which AI crawlers can read your site: ChatGPT, Perplexity, Claude, Google and Bing. Enter your website to start from your current robots.txt, or start from scratch.

Free, no sign-up. Nothing changes on your site until you upload the file.

Search and answers

These decide whether AI can find and cite you. Keep them allowed.

Visits for a user

When someone asks an AI about your page.

Model training

Your choice. Blocking these doesn't take you out of AI search answers.

robots.txt

Upload it as robots.txt at the root of your site. Each subdomain needs its own. OpenAI and Perplexity say changes take about a day.

How the rules work

Each crawler follows the group in robots.txt that names it. If none does, it follows the group for all crawlers, User-agent: *. That's why a group for each AI crawler is the clearest way to say what you want: it wins over your general rules.

Crawler kindIf you block it
Search crawlersThat AI can't find or cite your pages in its answers
Visits for a userThat AI may not open your page when someone asks about it
Training crawlersYour pages aren't used to train that company's future models. Search answers aren't affected

After you upload it

  1. 1

    Check it

    Run the AI crawler checker to make sure every rule reads the way you meant.

  2. 2

    Check your firewall

    A firewall, CDN or security plugin can block a crawler that robots.txt allows.

  3. 3

    Give it a day

    OpenAI and Perplexity say changes take about 24 hours to take effect.

Questions

Will this replace my whole robots.txt?

Only if you want it to. If you load your current file, we keep every rule that isn't about these AI crawlers and your Sitemap lines, and put the AI crawler rules on top. Read it through before you upload it.

Which AI crawlers should I allow?

Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot) if you want to be found and cited in AI answers. Training crawlers are your choice and don't affect search answers.

Does Perplexity-User follow robots.txt?

Perplexity says Perplexity-User, which fetches a page because a user asked about it, generally ignores robots.txt. It's in the list so your file states your intent.

My rules for all crawlers block everything. Does that block the AI crawlers?

Not the ones listed here. A crawler follows the group that names it, so a User-agent group for OAI-SearchBot with Allow: / wins over User-agent: * with Disallow: /.

Crawlers sorted. Now see what AI says.

The free AI check shows who AI recommends instead of you for your buyers' questions. About a minute, no sign-up.