Why your logs tell you more than robots.txt
robots.txt says what you'd like crawlers to do. Your server logs show what actually happened: which AI crawlers came, which pages they read, and whether your server let them in. A crawler that keeps getting 403, 429 or 503 is being turned away by your firewall, CDN or security plugin, whatever robots.txt says.
- 1
Download your access log
From your host's control panel, your CDN, or the server's access log file. A day or a week is plenty.
- 2
Paste or drop it here
It's read in your browser, line by line.
- 3
Look for blocks
Any AI search crawler with blocked answers, or missing entirely, is worth a look.
What the answers mean
| Answer | What it means for the crawler |
|---|---|
| 200, 301, 304 | It got the page, or was sent to the right one |
| 404 | The page doesn't exist. Fine now and then; a lot means broken links |
| 401, 403 | Turned away. Usually a firewall, bot protection or login |
| 429 | Rate limited: it asked too often and was told to slow down |
| 500 to 599 | Your server had an error, or a firewall answered for it |
Questions
Do my logs get uploaded anywhere?
No. The checker runs entirely in your browser. Your logs are read on your own device and never sent to Searchistry or anyone else.
Which log formats work?
The common Apache and Nginx access log formats, and logs with one JSON object per line, like many CDN and hosting logs. Files can be plain text or gzipped.
Why does a crawler get 403 or 429 when robots.txt allows it?
Your firewall, CDN, security plugin or rate limits are turning it away before robots.txt matters. That's one of the most common reasons a site is invisible to AI.
Can a user agent be faked?
Yes. Anyone can send a request that claims to be GPTBot. OpenAI and Perplexity publish the IP ranges their crawlers use, so you can check the IP addresses in your logs against them.
Why isn't Google-Extended in my logs?
Google-Extended isn't a separate crawler. It's a token you use in robots.txt; the crawling itself is done by Google's usual crawlers.