apprun.icuGuides

Guide · Developer tools

llms.txt and AI crawlers: what to publish and what to block

llms.txt tells a language model what your site is worth reading. robots.txt decides whether a crawler may fetch it at all. They are separate files solving separate problems, and only one of them is access control — and it is only advisory.

By apprun.icuUpdated Read 8 minIncludes a working browser tool
On this page
  1. The short answer
  2. What llms.txt is, and what it is not
  3. Where the file goes
  4. The difference between llms.txt and robots.txt
  5. Which AI crawlers exist, and what each one wants
  6. Why robots.txt is not access control
  7. How to check whether your domain is reachable
  8. What a good entry looks like
  9. Try it: read your llms.txt the way a person would
  10. Common questions
  11. Sources and further reading
Publish /llms.txt → crawlers read the map → readers get your wordsPublish /llms.txtCrawler readsthe mapReader getsyour words
Publish /llms.txt → a crawler that is allowed in reads the map → the reader gets an answer with your pages in it. Nothing in that chain requires you to hand over your content.

What llms.txt is, and what it is not

An llms.txt file is a markdown document you publish on your own site so a language model can find out what the site holds without crawling it blind. It is a curated map: a name, a short summary, and a list of links, each carrying one line saying what the reader will get if they follow it. The current proposal is version 2 of the specification at llmstxt.org.

The format is fixed order in markdown. An H1 carrying the project or site name is the only required element. After it come an optional blockquote summary, an optional details paragraph, then any number of H2 sections each holding a list of - [Title](url): note entries. The specification asks for concise language, a brief informative description on each link, and a file small enough to fit in a context window while the detail sits behind the links.

Two things it is not. It is not a permission: no crawler is obliged to read it, and ignoring it costs you nothing. And it is not a ranking signal — nothing in the specification promises one. The proposal is explicit that the information is used on demand, when an agent needs it while answering someone, and that its authors expected it to matter mainly for inference rather than training.

Where the file goes

At /llms.txt, the same root that holds /robots.txt and /sitemap.xml. The path carries meaning: a file at any subpath, such as /docs/llms.txt, covers every URL beneath it, and where more than one file applies an agent should use the most specific one. That rule is what lets a project living in a subdirectory of a shared host take part at all. The proposal considered and rejected the /.well-known/ prefix of RFC 8615 precisely because those URIs exist only at the origin root, which a path-only publisher can never reach.

Check yours the way you would check any file: request the URL and confirm a 200 with plain text rather than a redirect. Version 2 also recommends two standard link relations so an agent can find the file and the markdown versions of your pages without guessing: rel="alternate" type="text/markdown" and rel="describedby".

The difference between llms.txt and robots.txt

They answer different questions. robots.txt answers who may fetch; llms.txt answers what is worth reading once they may. A sitemap is a third thing: it lists every indexable page, rarely includes the markdown versions, and is too large to read as a whole, which is why the specification says it is not a substitute.

The practical consequence is ordering. Block a crawler and your llms.txt is never read, because nothing fetched the file containing the links. Allow it and the map is available.

Which AI crawlers exist, and what each one wants

Every token below is taken from that vendor’s own documentation, and each owner name links to the page it was taken from. Matching is case-insensitive against the product token the crawler sends in its own User-agent string, so a rule written for one vendor never covers another vendor’s bot.

The AI crawler tokens worth a decision
User-agent tokenOwnerWhat it fetches
GPTBotOpenAITraining corpus
ClaudeBotAnthropicTraining corpus
Google-ExtendedGoogleVertex grounding and Gemini training
Applebot-ExtendedAppleTraining corpus
meta-externalagentMetaTraining corpus
AmazonbotAmazonProduct data and training
CCBotCommon CrawlOpen research corpus
OAI-SearchBotOpenAIChatGPT search results
Claude-SearchBotAnthropicSearch and grounding
PerplexityBotPerplexitySearch index
GooglebotGoogleGoogle Search itself
ApplebotAppleSiri and Spotlight
Amzn-SearchBotAmazonAmazon search index
Meta-WebIndexerMetaIndexing and link previews
ChatGPT-UserOpenAIPages a user asks about
Claude-UserAnthropicPages a user asks about
Perplexity-UserPerplexityPages a user asks about
Amzn-UserAmazonAlexa and shopping answers
Meta-ExternalFetcherMetaPages a user asks about
OAI-AdsBotOpenAIAd campaign measurement
Meta-ExternalAdsMetaAd campaign measurement

Group them by purpose, because that is what the decision is about. Training crawlers collect pages for corpora; search and grounding crawlers produce results and citations; user-triggered fetchers only run when a person asks a model to read a page; ad bots check campaigns. A sensible default blocks the training and research tokens, leaves search and grounding open so you stay findable, and leaves user-triggered fetchers open because a person asked for your page by name.

Why robots.txt is not access control

RFC 9309 removes the ambiguity. The standard says of its own rules that “these rules are not a form of access authorization”. A crawler that chooses to honour them gets your preferences; a crawler that does not, fetches anyway. That is not a flaw in the standard, it is the only honest description of how the web works.

Several vendors say so about their own bots. OpenAI states that for ChatGPT-User “robots.txt rules may not apply”. Perplexity states that Perplexity-User “generally ignores robots.txt rules”. Meta documents that Meta-ExternalFetcher may bypass robots.txt and that the request carries the visitor’s IP address. By contrast Anthropic states its crawlers follow robots.txt and that site owners can control Claude-User, and Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”, so blocking it costs you nothing in search.

So the honest summary of a Disallow line is: this is what we would prefer. Anything that must genuinely stay private belongs behind a login. One more trap: RFC 9309 defines no Crawl-delay field at all, so a rate-limit line is not a guarantee either.

How to check whether your domain is reachable

Four checks, cheapest first. Request https://yourdomain.com/robots.txt and confirm a 200 and plain text; a redirect, a 404 or an HTML error page all read to a crawler as a missing file. Then find the User-agent group for your exact token and read it: Disallow: / blocks everything, Allow: / permits everything, Disallow: /admin blocks one area.

Resolve overlaps by hand, because the rule is short: the longest matching path wins, and where two rules match with the same length, Allow wins. That is what makes Disallow: /admin together with Allow: /admin/public behave as expected. Finally, confirm against reality rather than against the file: your own server logs show what actually arrived, and each vendor crawler page describes the reach of its own bot.

What a good entry looks like

Write the summary for someone who has never seen the site. One or two sentences that say what it is and who it is for beat a paragraph of positioning. Then group links under headings that reflect how a reader would ask about it: Docs, Guides, Pricing, Optional. A convention worth keeping is to put secondary material in a section named Optional so an agent working with a tight context can skip it.

Give every link a note that says what the page contains, not what the page is called. “Install the client and make a first request” tells an agent whether fetching is worth the call; “API quick start” does not. Use absolute https URLs, since a relative one breaks the moment the file is read away from your site, and keep titles unique. Finally, test the way the specification suggests: ask an agent a question your site answers, give it only your file as the starting point, and see whether it finds the right page.

Read a robots.txt like a crawler

Runs in your browser

Paste a robots.txt and see which AI crawlers it allows, which it blocks and which fall back to the wildcard group. Nothing is uploaded — the file is parsed in this tab.

Method: the file is split into user-agent groups; the decision shown is the one that applies to a request for /. A group naming the token is used when one exists; otherwise the User-agent: * group applies. Within those rules the longest match wins and Allow wins a tie (RFC 9309). A token with no matching group is allowed by default. This mirrors what a compliant crawler computes — it does not model crawlers that ignore robots.txt.

Try it: read your llms.txt the way a person would

An llms.txt is plain Markdown with a strict opening structure. Paste a draft below to see it rendered — headings, the blockquote summary and your file lists — which is the fastest way to spot a malformed list or a heading that drifted out of order before you publish it.

Try it with your own input

Your result

Common questions

Does llms.txt control whether AI bots can crawl my site?

No. It is the map you hand over once a crawler is already fetching your pages. Access is decided in robots.txt, one User-agent group per crawler product token.

Does llms-full.txt exist?

Not in the current specification. Version 1 described a tool for expanding the file into a full context; version 2 removed that context-expansion tooling, and with it the special meaning the old Optional section carried.

Can robots.txt actually block an AI crawler?

Only for crawlers that choose to honour it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anything that must stay private needs authentication.

Will blocking Googlebot hurt my search rankings?

Yes. Googlebot is Google Search: disallowing it removes your pages from the index. Google-Extended is the separate opt-out for training, and Google states it does not affect inclusion in Search or act as a ranking signal.

How many links should an llms.txt file contain?

Enough to navigate, few enough to read. The specification expects the file to fit in a model context while the detail sits behind the links, so a few dozen entries across a handful of clear sections is a sensible working size.

Does a Crawl-delay line slow crawlers down?

Do not rely on it. RFC 9309 defines no Crawl-delay field, so support is a vendor convention that some crawlers implement and others ignore.

Sources and further reading

Share this guide