1. Home
  2. Free tools
  3. AI crawler log checker

Free tool

AI crawler log checker

See which AI crawlers really visit your site, which pages they read, and whether your server lets them in. Paste or drop your access log: it never leaves your browser.

Check my logs

Or drop a file here. Plain text or .gz, up to 300 MB. Your logs stay in your browser.

Why your logs tell you more than robots.txt

robots.txt says what you'd like crawlers to do. Your server logs show what actually happened: which AI crawlers came, which pages they read, and whether your server let them in. A crawler that keeps getting 403, 429 or 503 is being turned away by your firewall, CDN or security plugin, whatever robots.txt says.

  1. 1

    Download your access log

    From your host's control panel, your CDN, or the server's access log file. A day or a week is plenty.

  2. 2

    Paste or drop it here

    It's read in your browser, line by line.

  3. 3

    Look for blocks

    Any AI search crawler with blocked answers, or missing entirely, is worth a look.

What the answers mean

AnswerWhat it means for the crawler
200, 301, 304It got the page, or was sent to the right one
404The page doesn't exist. Fine now and then; a lot means broken links
401, 403Turned away. Usually a firewall, bot protection or login
429Rate limited: it asked too often and was told to slow down
500 to 599Your server had an error, or a firewall answered for it

Questions

Do my logs get uploaded anywhere?

No. The checker runs entirely in your browser. Your logs are read on your own device and never sent to Searchistry or anyone else.

Which log formats work?

The common Apache and Nginx access log formats, and logs with one JSON object per line, like many CDN and hosting logs. Files can be plain text or gzipped.

Why does a crawler get 403 or 429 when robots.txt allows it?

Your firewall, CDN, security plugin or rate limits are turning it away before robots.txt matters. That's one of the most common reasons a site is invisible to AI.

Can a user agent be faked?

Yes. Anyone can send a request that claims to be GPTBot. OpenAI and Perplexity publish the IP ranges their crawlers use, so you can check the IP addresses in your logs against them.

Why isn't Google-Extended in my logs?

Google-Extended isn't a separate crawler. It's a token you use in robots.txt; the crawling itself is done by Google's usual crawlers.

Crawlers are visiting. Are they naming you?

The free AI check shows who AI recommends for your buyers' questions. About a minute, no sign-up.