llms.txt vs robots.txt: a standard that controls access, a proposal that offers a map

What is the difference between llms.txt and robots.txt?

One controls access; the other offers a map. robots.txt is the Robots Exclusion Protocol, published as RFC 9309 on the IETF Standards Track in September 2022. It lives at /robots.txt, it tells automated clients which paths they may fetch, and it places obligations on conforming crawlers: how to match rules, how long to cache the file, what to do when it cannot be fetched. Every major AI crawler operator documents that its crawlers follow it.

llms.txt is a proposal. Its site describes it as a proposal to standardise on a file that provides information to help agents use a website. It was first published by Jeremy Howard on 3 September 2024 and updated to a second version on 10 August 2026. The file is markdown: an H1 with the name of the site, an optional summary, and sections of links to the pages an agent would find most useful, with clean markdown versions of those pages suggested alongside. It allows nothing and forbids nothing.

What does our own data say about each file?

We publish an llms.txt and a markdown twin of each page on attensira.com, because they cost almost nothing to generate. In our AI traffic snapshot for the 30 days to 4 October 2026, neither /llms.txt nor /robots.txt was among the 30 paths AI crawlers fetched most; the 30th, /privacy, had 2,365 requests. This comparison page itself was fetched 53 times by AI crawlers in that window. A file nobody asks for often is not a file an engine is leaning on.

The answers point the same way. We track "how to block AI crawlers in robots.txt" every day. ChatGPT's stored answer of 4 October 2026 is entirely about robots.txt groups and never mentions llms.txt, and across 33 ChatGPT answers to that prompt in 30 days, about 82% of 1,507 citations went to three operators' own documentation: developers.google.com, anthropic.com and help.openai.com. On access, engines defer to the operator's page.

How do large sites use robots.txt for AI?

Deliberately, and differently for training and search. On 3 September 2026 we read /robots.txt on the top 10,000 domains in a dated Tranco list: 8,036 answered with a file or an authoritative 404, and 396 (4.7%) failed with a timeout, a server error or a rate limit. 1,147 (14.3%) disallow GPTBot, OpenAI's training crawler, at the root, while only 467 (5.8%) disallow OAI-SearchBot, the crawler behind ChatGPT search. Large sites use robots.txt to keep training out and search in.

In the same run we probed /llms.txt wherever a site's own robots.txt let our agent read it. 966 of 7,781 domains (12.4%) served one. That is real adoption for a two-year-old proposal, though our study does not say which kinds of sites publish one or whether any engine reads them. The full method and every crawler row are on our study page.

What does RFC 9309 actually require?

A few things that matter in practice:

The failure modes are the expensive part. A robots.txt that returns a server error during a deploy tells conforming crawlers to stay out of the whole site. One that returns a 404 after a migration tells them everything is allowed. Neither shows up in your analytics. Monitor the status code your robots.txt returns the way you monitor your home page, and alert on anything other than a 200, because a single bad deploy can close the whole site to every conforming crawler for as long as the error lasts.

How should an llms.txt be built if you publish one?

Generated, never hand-written. The file is only useful if it lists pages that exist and are current, and a hand-maintained list starts to rot the week after it is written. Generate it from the same source that builds your sitemap or navigation, so a page that is removed disappears from the file in the same deploy. Keep it short: the proposal's point is curation, so list the pages an agent should read first, not every URL you have.

If you also publish markdown versions of pages, as the proposal suggests, make sure they say the same thing as the HTML pages. A markdown twin that drifts from its page is worse than none, because an agent reading it will repeat facts the page no longer states. Render them from the same source as the pages for that reason.

What happens when the two files disagree?

robots.txt wins, for every crawler that follows it:

• Lists the page

  robots.txt

  Disallows the path

  What a conforming crawler does

  Does not fetch it

• Omits the page

  robots.txt

  Allows the path

  What a conforming crawler does

  Can still find it through links and sitemaps

• Lists the page

  robots.txt

  Allows the path

  What a conforming crawler does

  May fetch it; the listing changes nothing it must do

The only way the two can conflict in a costly way is by misleading the people who maintain them: a team that sees a page listed in llms.txt may assume it is readable by AI engines when robots.txt says otherwise. Check both whenever you change either.

What would change our view?

Documentation. If OpenAI, Perplexity, Anthropic or Google publish that their answer engines read llms.txt when choosing sources, or if a controlled study shows sites with the file being cited more often than comparable sites without it, the file moves from convenience to requirement. Until then, the honest position is the one Google states for its own AI features: no AI text file is needed. We will revisit this page when any of those vendors changes its crawler documentation, and the date at the top of the page tells you when we last checked.

What can you do today?

Fetch your live robots.txt from every host you serve. Search it for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot and confirm none is disallowed by accident, including through a wildcard group. Then look at a week of edge logs to see which of those crawlers actually arrived. Only after that, decide whether llms.txt is worth an afternoon. If your platform can generate it automatically, the answer is probably yes; if it means hand-maintaining a list, spend the afternoon on the pages instead.