llms.txt and AI crawlers: who you let read your site

In the robots.txt files of most sites we look at, the word GPTBot doesn’t appear at all. Where it does appear, it’s usually on a Disallow: / line added after one of those “protect your content from AI” articles. That line keeps your content out of model training. But sites that block “all AI bots” wholesale also close the door that ChatGPT, Claude and Perplexity use to find pages to recommend to their users.

This article sorts AI crawlers by what they actually do, based on each provider’s own documentation as of September 2026. You get two copy-paste robots.txt files, one for an e-shop and one for a services site, plus a sample llms.txt. We also cover what llms.txt does and doesn’t do, because that’s where most of the hype is.

The short answer

• There is no single “AI crawler”. Most providers run separate bots for training, for a search index, and for fetching a page when a user asks. You can block each one separately.
• If you want to show up in AI answers, keep the search and retrieval bots open: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. And Googlebot, of course, which AI Overviews depend on.
• Training is your call: GPTBot, ClaudeBot, CCBot, Amazonbot, Google-Extended. Blocking them doesn’t cost you your presence in those tools’ search.
• robots.txt is a request, not a wall. User-triggered fetches may ignore it. A real block takes a firewall rule.
• No major crawler officially reads llms.txt. It’s cheap and harmless, but it won’t get you into AI answers.

Three jobs, not one crawler

When an AI tool answers a question about your business, the information got there by one of three routes:

1. Training. At some point a bot read your page and it became part of the data the model was trained on. That knowledge is frozen at training time, so prices and stock go stale quickly.
2. Search index. A bot builds a catalogue of pages, the way Googlebot does, so the tool can find and cite sources when it searches the web.
3. Retrieval at question time. A user asks something, the tool opens your page right then, and answers with fresh information, usually with a link.

Routes 2 and 3 bring citations and visits. Route 1 makes the model “know” you, but with a lag and without a link. It’s the distinction from our piece on what GEO is, translated into robots.txt lines.

Which crawler does what

From the official pages of OpenAI, Anthropic, Perplexity, Google, Common Crawl and Amazon:

User-agent	Provider	Job	Respects robots.txt?	If you block it
GPTBot	OpenAI	Training	Yes	Your content isn’t used for model training
OAI-SearchBot	OpenAI	Search index	Yes	You don’t appear in ChatGPT search answers
ChatGPT-User	OpenAI	Retrieval at question time	”May not apply”	The rule may be ignored, because a user requested the fetch
ClaudeBot	Anthropic	Training	Yes, plus Crawl-delay	Future content is excluded from training
Claude-SearchBot	Anthropic	Search index	Yes, plus Crawl-delay	You’re not indexed for Claude’s search
Claude-User	Anthropic	Retrieval at question time	Yes, plus Crawl-delay	Claude won’t open your page when a user asks
PerplexityBot	Perplexity	Search index (not training)	Yes	You don’t appear as a source in Perplexity
Perplexity-User	Perplexity	Retrieval at question time	Generally no	The rule is ignored, because a user requested the fetch
Google-Extended	Google	Gemini training and grounding	Yes (not a separate bot, it’s a rule applied to Googlebot’s crawl)	You’re out of Gemini training and grounding. Google Search is unaffected
CCBot	Common Crawl	Open web archive, widely used as training data	Yes	You’re left out of new Common Crawl archives
Amazonbot	Amazon	Improving services, possible model training	Yes, no Crawl-delay	Your content isn’t used to train Amazon’s models
Bytespider	ByteDance	No reachable official documentation	Unknown	You can’t verify it

Two points trip most people up. First, Google says that appearing in AI Overviews and AI Mode is controlled with the usual Search tools (noindex, nosnippet, max-snippet), not with Google-Extended. Second, Amazon runs separate bots for search and for user requests (Amzn-SearchBot, Amzn-User), and neither is used for training.

What “blocking” means in practice

Before you copy any snippet, keep four things in mind:

• Blocking isn’t retroactive. Blocking GPTBot today doesn’t pull out what it has already read. It applies to whatever gets crawled from now on.
• User-triggered fetches are a grey area. OpenAI and Perplexity say openly that ChatGPT-User and Perplexity-User may not follow robots.txt, because a person asked for the page.
• robots.txt is a request, not a security mechanism. Anyone who doesn’t want to honour it won’t. For a real block you need a firewall or CDN rule. Most CDN providers now offer a ready-made AI bot option.
• A bot with its own group ignores the User-agent: * group. That’s the rule in the RFC 9309 standard: a crawler follows only the most specific group that matches it. If you add “User-agent: OAI-SearchBot / Allow: /” to welcome it in, you drop every Disallow for your cart, checkout and filters for that bot. That’s why the snippets below don’t name the bots we want open. They’re left to follow *.

That last point isn’t theoretical. The e-shop that runs our own chatbot uses the default PrestaShop robots.txt: 96 Disallow lines under a single User-agent: *, and no mention of any AI bot. A single group for one specific bot is enough to wipe out all 96 lines for that bot.

robots.txt for an e-shop

The logic: open for search and retrieval, so AI tools see current prices and availability. Closed for bots that only train. For a shop, training offers little: prices in the model go stale, and thousands of product and filter URLs add load to your server without bringing visits.

# robots.txt — e-shop
# Search and retrieval bots (Googlebot, OAI-SearchBot,
# ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot,
# Amzn-SearchBot) are deliberately NOT named, so they
# follow the "User-agent: *" rules.

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /my-account
Disallow: /search
Disallow: /*?order=
Disallow: /*?q=

# Training-only bots: no visits back to a shop
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /

Sitemap: https://www.example.com.cy/sitemap.xml

Adapt the paths to your platform. On PrestaShop and WooCommerce, keep the rules you already have under * and add only the second block. We left Google-Extended open on purpose, because it also controls grounding for Gemini’s answers, not just training. If you’d rather the models themselves know your shop, delete the GPTBot and ClaudeBot lines.

robots.txt for a services site

For a services company, an office or a local trade, the maths flips. The content doesn’t change daily, and a model “knowing” you when someone asks “who does X in Limassol” works in your favour. Here we leave almost everything open:

# robots.txt — services site
User-agent: *
Allow: /
Disallow: /admin

# No reachable official documentation from the operator
User-agent: Bytespider
Disallow: /

Sitemap: https://www.example.com.cy/sitemap-index.xml

This is also how pythia.works is set up: Allow: / for everyone, plus the sitemap. Our robots.txt is generated at build time from the same setting that puts the site into “Coming Soon” mode, and so are the sitemap and the llms.txt. While the site is under construction, robots.txt blocks everything (Disallow: /) and llms.txt carries only the title. When the site opens, all three open with it. That way a half-finished site can’t be left open by mistake, and a finished one can’t be left closed. One more thing: robots.txt is a public file, so don’t put paths you want kept secret in it.

llms.txt: what it is, who reads it, is it worth it?

llms.txt was proposed by Jeremy Howard in September 2024. It’s a markdown file at the root of your site with a short description and your most useful links. The second version of the proposal came out in August 2026. The proposal itself says it’s meant for use at question time, not for training.

Where things stand today, without the hype:

• None of the crawlers in the table says in its documentation that it reads other sites’ llms.txt.
• Google says explicitly that AI Overviews and AI Mode need no “new machine readable files, AI text files, or markup”.
• On the other hand, Chrome checks for it. Lighthouse now has an “Agentic Browsing” category with an llms.txt audit. It only flags a problem if the server returns an error. If the file is simply missing, the audit is marked “Not Applicable”, not failed.
• The providers publish it themselves for their documentation. OpenAI, Anthropic and Perplexity all have an llms.txt on their docs. In other words, agents and developer tools use it, not search.

In our practical guide to AI search we put llms.txt on the list, and it stays there: it takes half an hour, costs nothing and breaks nothing. But if someone sells it to you as the reason ChatGPT will recommend you, they’re not telling you the truth. What matters there is open search bots and a site that reads without JavaScript.

Here’s a sample for a fictional services business. All details are examples:

# Example Physiotherapy Clinic

> Physiotherapy clinic in Limassol with two certified
> physiotherapists. Sports injuries, post-surgery rehabilitation
> and lower back pain. Book online or by phone.

- Address: 1 Example Street, 3000 Limassol
- Hours: Monday–Friday 08:00–19:00
- Languages: Greek, English

## Services

- [Sports injuries](https://www.example.com.cy/en/sports-injuries): assessment and rehab programme
- [Post-surgery rehabilitation](https://www.example.com.cy/en/rehabilitation): knee, shoulder, hip
- [Prices](https://www.example.com.cy/en/prices): session and package price list

## Contact

- [Book an appointment](https://www.example.com.cy/en/booking): online booking
- [FAQ](https://www.example.com.cy/en/faq): insurance, GESY, cancellations

## Optional

- [Blog](https://www.example.com.cy/en/blog): exercises and advice

The structure follows the proposal: an # title, a blockquote summary, free text, and ## sections with lists of markdown links. The ## Optional section is the “if you have time” part. Watch the links: the proposal specifies that every line in the lists is a markdown link ([text](url)), not a bare URL.

Two things we learned from our own llms.txt. First, a hand-written llms.txt goes stale. Our original one, written by hand, was already missing two of the blog’s five articles. Prices go the same way: every price you put in it is one more place you have to remember to update. So either keep it price-free and link to your pricing page, as the sample does, or generate it automatically from the same content as the site, which is what we do now. Second, on a bilingual site, publish one file per language, at /llms.txt and /en/llms.txt, with a link from each to the other.

What to do tomorrow

1. Open yoursite.com/robots.txt and check whether it mentions any AI bot. If it doesn’t, they all follow *. That’s usually fine, it just wasn’t a conscious choice.
2. If you find “block all AI”, check that you haven’t shut out OAI-SearchBot, Claude-SearchBot and PerplexityBot without meaning to.
3. Decide on training using the logic above: a shop whose prices change, probably closed. A services business, probably open.
4. Add an llms.txt if you have half an hour spare, with no ranking expectations.

If you’re building a new site, add robots.txt and llms.txt to the “what’s included in the SEO” question, one of the five we suggest in our piece on how much a website costs in Cyprus. It’s a few minutes’ work for someone who knows it. That’s also the thinking behind our own website design, where on-page SEO is standard in every project and isn’t charged separately. If you’d like someone to look at your current site this way (robots.txt, sitemap, structured data, what AI bots actually see), that’s what our SEO/GEO audits are for. Send us your URL.