Your robots.txt matters far more than llms.txt. Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot), decide separately about the training crawlers (GPTBot, ClaudeBot, Google-Extended), and check your firewall isn't blocking any of them. llms.txt is a proposed standard that's cheap to add, but Google has said it doesn't use it.
Search crawlers and training crawlers
AI companies run different crawlers for different jobs. Search crawlers fetch pages so the AI can find and cite them in answers. Training crawlers collect pages to train future models. Blocking a training crawler doesn't remove you from that company's search answers.
| Crawler | Company | Job |
|---|---|---|
| OAI-SearchBot | OpenAI | Search: pages shown in ChatGPT search answers |
| GPTBot | OpenAI | Training |
| PerplexityBot | Perplexity | Search: pages shown and linked in Perplexity |
| Claude-SearchBot | Anthropic | Search: helps Claude's search find pages |
| ClaudeBot | Anthropic | Training |
| Googlebot | Search, including AI Overviews and AI Mode | |
| Google-Extended | A robots.txt token for Gemini training and grounding outside Search. No effect on Search | |
| ChatGPT-User, Claude-User, Perplexity-User | OpenAI, Anthropic, Perplexity | Visit a page when a user asks the AI to |
Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. OpenAI separates GPTBot (training) from OAI-SearchBot (search). So you can opt out of training and still be found.
Build your AI crawler rules
Tick the crawlers you want to allow. The rules update as you go: copy them into the robots.txt at the root of your site, above or below your existing rules.
Search and answers
Leave these on to be found and cited.
Fetches for a user
Visits when someone asks about your page.
Model training
Your choice: blocking these doesn't remove you from search answers.
Add your usual rules and your sitemap line below it. Changes can take about a day to reach ChatGPT and Perplexity.
Anthropic notes that robots.txt applies per subdomain, so blog.yoursite.com needs its own file, and that its crawlers support the Crawl-delay rule if you want to slow them down.
Check your firewall, not just robots.txt
robots.txt is a request. A firewall, CDN or security plugin can block a crawler outright, whatever robots.txt says, and it's one of the most common reasons a site is invisible to AI.
- 1
Look at your firewall's bot settings
Many CDNs and security plugins have a setting for AI bots. Make sure the search crawlers aren't on a block list.
- 2
Allow by user agent and IP
Perplexity, for one, publishes the IP ranges its crawlers use and recommends allowing them by both.
- 3
Check your server logs
Search for the user agents above. If a crawler never appears, something is stopping it.
- 4
Give it a day
OpenAI and Perplexity say changes take about 24 hours to take effect.
What llms.txt is
llms.txt is a proposal by Jeremy Howard of Answer.AI, published on September 3, 2024: a Markdown file at /llms.txt that gives AI tools a short summary of a site and links to its most useful pages (llmstxt.org).
# Your Brand
> One sentence on what you do and who it's for.
## Product
- [Pricing](https://yoursite.com/pricing/): plans and prices
- [How it works](https://yoursite.com/product/): the main features
## Optional
- [Terms](https://yoursite.com/terms/)It's a proposal, not a standard the big AI companies have adopted. Google is the clearest: it doesn't use it for Search.
“Gary Illyes clearly stated that Google doesn't support LLMs.txt and isn't planning to”
Kenichi Suzuki, reporting Google's Gary Illyes, via Search Engine Land
Should you add one?
It takes an hour, does no harm, and some AI tools and coding assistants read it. Just don't expect it to change what ChatGPT or Google's AI say about you. We have one at searchistry.ai/llms.txt, built from the site itself so it never drifts.
Do
- Add one if it's cheap: a summary and links to your key pages
- Keep it accurate and in step with your site
- Fix robots.txt and your firewall first
Don't
- Expect it to raise your AI visibility on its own
- Put information in it that isn't on your pages
- Block crawlers and rely on llms.txt instead
Your crawler checklist
- 1
Allow the search crawlers
OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot.
- 2
Decide on training
GPTBot, ClaudeBot and Google-Extended: allow or block, knowing it doesn't affect search answers.
- 3
Check the firewall
No AI search crawler blocked by your CDN, firewall or plugins.
- 4
Confirm in your logs
See each crawler visit within a few days.
- 5
Add llms.txt if you like
A short summary and your key links.
Your checklist
0 of 6 doneTicks are saved in this browser only.
Questions people ask
Does blocking GPTBot hurt my visibility in ChatGPT?
Not in ChatGPT's search answers. OpenAI says those depend on OAI-SearchBot. GPTBot collects pages for training future models.
Do I need llms.txt to appear in Google's AI Overviews?
No. Google says no special file is needed, and Google doesn't use llms.txt for Search.
How do I know if AI crawlers visit my site?
Search your server or CDN logs for their user agents, such as OAI-SearchBot, PerplexityBot or Claude-SearchBot. If one never appears, check robots.txt and your firewall.
What is llms-full.txt?
A common companion file, not part of the original proposal, that puts the full text of a site's key pages in one Markdown file so an AI tool can read it in one request.
Should I block AI training crawlers?
That's a business decision about how your content is used. It doesn't decide whether AI search can find and cite you, as long as the search crawlers are allowed.
Sources
Checked on November 11, 2026. Numbers about AI answers come from our research; everything else from the companies' own pages.
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google: common crawlers, including Google-Extended
- Google Search Central: AI features and your website
- llmstxt.org: the /llms.txt file proposal
- Answer.AI: /llms.txt, a proposal to provide information to help LLMs use websites (September 3, 2024)
- Search Engine Land: Google says normal SEO works for AI Overviews and llms.txt won't be used