1. Home
  2. Guides
  3. llms.txt and AI crawlers

Set up and measure

llms.txt and AI crawlers: what to set up, and what to skip

Before any AI can recommend you, its crawler has to be able to read your site. Here's which crawlers matter, how to set them up in a few minutes, and where llms.txt fits.

  • 8 minute read
  • Updated November 11, 2026
  • 8 sources checked
The short answer

Your robots.txt matters far more than llms.txt. Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot), decide separately about the training crawlers (GPTBot, ClaudeBot, Google-Extended), and check your firewall isn't blocking any of them. llms.txt is a proposed standard that's cheap to add, but Google has said it doesn't use it.

Search crawlers and training crawlers

AI companies run different crawlers for different jobs. Search crawlers fetch pages so the AI can find and cite them in answers. Training crawlers collect pages to train future models. Blocking a training crawler doesn't remove you from that company's search answers.

From each company's crawler documentation, listed under Sources.
CrawlerCompanyJob
OAI-SearchBotOpenAISearch: pages shown in ChatGPT search answers
GPTBotOpenAITraining
PerplexityBotPerplexitySearch: pages shown and linked in Perplexity
Claude-SearchBotAnthropicSearch: helps Claude's search find pages
ClaudeBotAnthropicTraining
GooglebotGoogleSearch, including AI Overviews and AI Mode
Google-ExtendedGoogleA robots.txt token for Gemini training and grounding outside Search. No effect on Search
ChatGPT-User, Claude-User, Perplexity-UserOpenAI, Anthropic, PerplexityVisit a page when a user asks the AI to

Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. OpenAI separates GPTBot (training) from OAI-SearchBot (search). So you can opt out of training and still be found.

Build your AI crawler rules

Tick the crawlers you want to allow. The rules update as you go: copy them into the robots.txt at the root of your site, above or below your existing rules.

Search and answers

Leave these on to be found and cited.

Fetches for a user

Visits when someone asks about your page.

Model training

Your choice: blocking these doesn't remove you from search answers.

Your robots.txt

Add your usual rules and your sitemap line below it. Changes can take about a day to reach ChatGPT and Perplexity.

Anthropic notes that robots.txt applies per subdomain, so blog.yoursite.com needs its own file, and that its crawlers support the Crawl-delay rule if you want to slow them down.

Check your firewall, not just robots.txt

robots.txt is a request. A firewall, CDN or security plugin can block a crawler outright, whatever robots.txt says, and it's one of the most common reasons a site is invisible to AI.

  1. 1

    Look at your firewall's bot settings

    Many CDNs and security plugins have a setting for AI bots. Make sure the search crawlers aren't on a block list.

  2. 2

    Allow by user agent and IP

    Perplexity, for one, publishes the IP ranges its crawlers use and recommends allowing them by both.

  3. 3

    Check your server logs

    Search for the user agents above. If a crawler never appears, something is stopping it.

  4. 4

    Give it a day

    OpenAI and Perplexity say changes take about 24 hours to take effect.

What llms.txt is

llms.txt is a proposal by Jeremy Howard of Answer.AI, published on September 3, 2024: a Markdown file at /llms.txt that gives AI tools a short summary of a site and links to its most useful pages (llmstxt.org).

llms.txt (example)
# Your Brand

> One sentence on what you do and who it's for.

## Product

- [Pricing](https://yoursite.com/pricing/): plans and prices
- [How it works](https://yoursite.com/product/): the main features

## Optional

- [Terms](https://yoursite.com/terms/)

It's a proposal, not a standard the big AI companies have adopted. Google is the clearest: it doesn't use it for Search.

“Gary Illyes clearly stated that Google doesn't support LLMs.txt and isn't planning to”

Kenichi Suzuki, reporting Google's Gary Illyes, via Search Engine Land

Should you add one?

It takes an hour, does no harm, and some AI tools and coding assistants read it. Just don't expect it to change what ChatGPT or Google's AI say about you. We have one at searchistry.ai/llms.txt, built from the site itself so it never drifts.

Do

  • Add one if it's cheap: a summary and links to your key pages
  • Keep it accurate and in step with your site
  • Fix robots.txt and your firewall first

Don't

  • Expect it to raise your AI visibility on its own
  • Put information in it that isn't on your pages
  • Block crawlers and rely on llms.txt instead

Your crawler checklist

  1. 1

    Allow the search crawlers

    OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot.

  2. 2

    Decide on training

    GPTBot, ClaudeBot and Google-Extended: allow or block, knowing it doesn't affect search answers.

  3. 3

    Check the firewall

    No AI search crawler blocked by your CDN, firewall or plugins.

  4. 4

    Confirm in your logs

    See each crawler visit within a few days.

  5. 5

    Add llms.txt if you like

    A short summary and your key links.

Your checklist

0 of 6 done

Ticks are saved in this browser only.

Questions people ask

Does blocking GPTBot hurt my visibility in ChatGPT?

Not in ChatGPT's search answers. OpenAI says those depend on OAI-SearchBot. GPTBot collects pages for training future models.

Do I need llms.txt to appear in Google's AI Overviews?

No. Google says no special file is needed, and Google doesn't use llms.txt for Search.

How do I know if AI crawlers visit my site?

Search your server or CDN logs for their user agents, such as OAI-SearchBot, PerplexityBot or Claude-SearchBot. If one never appears, check robots.txt and your firewall.

What is llms-full.txt?

A common companion file, not part of the original proposal, that puts the full text of a site's key pages in one Markdown file so an AI tool can read it in one request.

Should I block AI training crawlers?

That's a business decision about how your content is used. It doesn't decide whether AI search can find and cite you, as long as the search crawlers are allowed.

Sources

Checked on November 11, 2026. Numbers about AI answers come from our research; everything else from the companies' own pages.

  1. OpenAI: Overview of OpenAI crawlers
  2. Perplexity: Perplexity crawlers
  3. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  4. Google: common crawlers, including Google-Extended
  5. Google Search Central: AI features and your website
  6. llmstxt.org: the /llms.txt file proposal
  7. Answer.AI: /llms.txt, a proposal to provide information to help LLMs use websites (September 3, 2024)
  8. Search Engine Land: Google says normal SEO works for AI Overviews and llms.txt won't be used

Check that AI can read your site

The free AI check shows what AI says about you today. In the dashboard, the crawlability check shows what's stopping AI from reading your site.