What this checks
- robots.txt, crawler by crawlerWhether each AI crawler may read your site, and which rule decides it.
- llms.txtWhether you have one, and what it starts with.
- Your sitemapWhether crawlers can find it, and how many pages it lists.
AI platforms run different crawlers for different jobs. The search crawlers decide whether AI can find and cite you. The training crawlers collect pages for future models, and blocking them doesn't take you out of AI search answers. The llms.txt and AI crawlers guide explains each one.
How to read the results
| Result | What it means |
|---|---|
| Allowed | No rule stops this crawler from reading the page |
| Allowed, some paths blocked | It can read the page you checked, but some other addresses on your site are blocked for it |
| Blocked | A rule tells this crawler not to read the page |
When no group in your robots.txt names a crawler, it follows the rules for all crawlers (User-agent: *). A missing robots.txt means everything is allowed. If robots.txt can't be loaded at all, crawlers may treat the whole site as blocked until it loads again.
Questions
How does the AI crawler checker work?
It reads your robots.txt, llms.txt and sitemap, the same public files any crawler reads, and works out what each AI crawler is allowed to do the way the robots.txt standard says crawlers should: the group that names the crawler, or the * group if none does, and the longest matching rule.
If the checker says a crawler is allowed, can it definitely reach my site?
Not always. robots.txt is a request. A firewall, CDN or security plugin can still block a crawler, and this checker can't see those settings. Check your firewall's bot settings and your server logs as well.
Should I block AI training crawlers?
It's your choice. Blocking GPTBot, ClaudeBot or Google-Extended doesn't remove you from AI search answers, which depend on the search crawlers such as OAI-SearchBot, PerplexityBot and Googlebot.
How long do robots.txt changes take to work?
OpenAI and Perplexity say about 24 hours. Run the checker again after you change your file to make sure the rules read the way you meant.
Do you store the sites I check?
We keep the answer for 10 minutes so a second check is quick, then it's gone. We don't save which sites were checked to any list or account.