1. Home
  2. Free tools
  3. AI crawler checker

Free tool

AI crawler checker

See which AI crawlers your robots.txt lets in: ChatGPT, Perplexity, Claude, Google and Bing, crawler by crawler. Plus whether you have an llms.txt and a sitemap.

Free, no sign-up. We read your public robots.txt, llms.txt and sitemap.

What this checks

  • robots.txt, crawler by crawlerWhether each AI crawler may read your site, and which rule decides it.
  • llms.txtWhether you have one, and what it starts with.
  • Your sitemapWhether crawlers can find it, and how many pages it lists.

AI platforms run different crawlers for different jobs. The search crawlers decide whether AI can find and cite you. The training crawlers collect pages for future models, and blocking them doesn't take you out of AI search answers. The llms.txt and AI crawlers guide explains each one.

How to read the results

ResultWhat it means
AllowedNo rule stops this crawler from reading the page
Allowed, some paths blockedIt can read the page you checked, but some other addresses on your site are blocked for it
BlockedA rule tells this crawler not to read the page

When no group in your robots.txt names a crawler, it follows the rules for all crawlers (User-agent: *). A missing robots.txt means everything is allowed. If robots.txt can't be loaded at all, crawlers may treat the whole site as blocked until it loads again.

Questions

How does the AI crawler checker work?

It reads your robots.txt, llms.txt and sitemap, the same public files any crawler reads, and works out what each AI crawler is allowed to do the way the robots.txt standard says crawlers should: the group that names the crawler, or the * group if none does, and the longest matching rule.

If the checker says a crawler is allowed, can it definitely reach my site?

Not always. robots.txt is a request. A firewall, CDN or security plugin can still block a crawler, and this checker can't see those settings. Check your firewall's bot settings and your server logs as well.

Should I block AI training crawlers?

It's your choice. Blocking GPTBot, ClaudeBot or Google-Extended doesn't remove you from AI search answers, which depend on the search crawlers such as OAI-SearchBot, PerplexityBot and Googlebot.

How long do robots.txt changes take to work?

OpenAI and Perplexity say about 24 hours. Run the checker again after you change your file to make sure the rules read the way you meant.

Do you store the sites I check?

We keep the answer for 10 minutes so a second check is quick, then it's gone. We don't save which sites were checked to any list or account.

Crawlers let in. Is AI naming you?

The free AI check asks AI the questions your buyers ask and shows who it recommends instead of you. About a minute, no sign-up.