llms.txt and where it actually stands

Is llms.txt worth publishing, and do crawlers read it?

Publishing one is cheap and low-risk, and there is no evidence in operator documentation that any crawler reads it. llms.txt is a public proposal — not a standard — for a Markdown file at a site's root that helps agents use the site. Treat it as a well-made index for the agents that choose to look, not as a visibility lever.

What the proposal actually says

llmstxt.org describes itself as a proposal to standardise on using an /llms.txt file to provide information to help agents use a website. It was authored by Jeremy Howard and published on 3 September 2024.[^llmstxt-definition]

The problem it identifies is specific and worth quoting, because it explains the file's shape better than any summary. Context windows, the proposal says, while larger than they were, are still too small for most websites — and converting a page built for humans back into clean text is difficult and imprecise.[^llmstxt-problem] The file is a response to both: a short, curated, already-clean entry point, rather than an exhaustive machine index.

The format is deliberately minimal.[^llmstxt-format]

There is one convention worth knowing: a section headed Optional is used, by convention, for secondary information — links an agent can skip when a shorter context is needed.[^llmstxt-optional] That is the proposal's one concession to budgeting, and it is the part most published files ignore.

A minimal conforming file is genuinely small.

# Example Corp

> Example Corp builds inventory software for independent retailers.
> The docs below cover setup, the API, and billing.

## Docs

- Getting started: install and first sync
- API reference: endpoints, auth, rate limits

## Optional

- Changelog: release history

Who has adopted it

Documentation publishers, broadly and visibly. The clearest way to check is to fetch one rather than trust a list, and Mistral's documentation site serves a conforming file at docs.mistral.ai/llms.txt: an H1 naming the project, an H2 Docs section, and Markdown links to documentation pages with an inline description after each.[^mistral-llmstxt-live]

That pattern — a documentation site, a flat list of pages, one line of description each — is the shape adoption has actually taken. Documentation platforms generate these files automatically, which is why the adopters skew so heavily toward products whose docs are hosted on such a platform rather than toward the web generally.

It is worth being precise about what that adoption demonstrates. It shows that publishers, including AI companies publishing their own documentation, find the format useful for making their material easy to consume. It does not show that anybody's crawler goes looking for the file on your site. Those are two different propositions, and the second is the one people care about.

The honest finding on whether crawlers read it

Every operator page used to build this guide was read for a mention of llms.txt. OpenAI's crawlers documentation, Anthropic's crawler support article, Perplexity's bots guide, Google's common crawlers reference, Apple's Applebot page, Amazon's Amazonbot page and Common Crawl's CCBot page: none of them mentions llms.txt at all.

That is a statement about what those pages say on 3 September 2026, and it is carefully bounded. It is not a claim that no system anywhere ever fetches the file. Agents built on top of these models can and do fetch whatever their authors tell them to, and a well-formed llms.txt is exactly the sort of thing a retrieval-augmented agent might be pointed at. What can be said is that no operator has documented a commitment, which means anyone telling you llms.txt gets you cited is describing a hope rather than a documented behaviour.

Contrast the situation with robots.txt, whose behaviour is specified in RFC 9309 and whose observance every operator in this guide states explicitly. That is what a documented commitment looks like, and it is the standard llms.txt has not yet met.

What llms.txt is not

Three confusions do real damage, and all three come from treating the file as something it does not claim to be.

It is not a sitemap. A sitemap enumerates every URL you want discovered and is consumed by crawlers that document consuming it. An llms.txt is short, curated and descriptive by design — the proposal exists precisely because exhaustive machine indexes are the wrong shape for a context window. Publishing an llms.txt listing four hundred pages misses the point of the format.

It is not an access control. It carries no permissions. Nothing in it can prevent a fetch, restrict training use, or attach a licence, and no operator has undertaken to honour any instruction placed there. Access decisions live in robots.txt and in your origin configuration, and a restriction written into llms.txt is a sentence with no enforcement behind it.

It is not a ranking factor. No operator documents reading it, so no operator documents weighting it. Anyone presenting llms.txt as a citation tactic is several inferences past the evidence.

How llms.txt compares to the files that already do this job

Sites already publish several machine-readable files, and llms.txt is easiest to place by seeing what each of the others answers.

Reading down the last column is the fastest way to prioritise. The first three rows have a documented consumer; the fourth does not. That is not an argument against publishing an llms.txt — it is an argument against publishing one instead of getting the first three right.

There is also a real overlap with the third row. The work of writing one honest sentence about what each important page is for is genuinely useful, and it improves the page itself, its description, and its structured data at the same time. If producing an llms.txt is what finally forces that exercise, the exercise is worth more than the file.

The variants, and a caution about them

The proposal has attracted companion conventions — most commonly an expanded file holding full page text rather than links, published alongside the index. These are not part of what llmstxt.org specifies, and a claim that any particular variant is read by any particular system should be treated exactly like the core claim: ask which operator documented it, and check the date.

The general caution is the useful part. This is an area where a plausible convention can circulate widely enough to look established without any operator ever having committed to it, and where the cost of believing the wrong thing is misdirected effort rather than an error message. When something in this space sounds settled, the question to ask is not whether it is popular but whose documentation says so.

Should you publish one?

A short cost-benefit, stated as a tradeoff rather than a recommendation.

The defensible position: publish one if it is nearly free, keep it short and accurate, and do not count it as progress on AI visibility. If producing it competes for time with making sure the citation crawlers can reach your pages at all, the crawlers win — that mechanism is documented, and this one is not.

If you do publish, three rules keep it honest. Keep it short enough that a person could read it. Point every link at a page that exists and returns a 200. And put a review of it on the same schedule as your robots.txt, because an index that describes last year's site is a liability rather than an asset.

One last practical note. Because the file is Markdown, it is readable by a person, which makes it unusually easy to keep honest compared with a generated XML sitemap. Use that: if the description beside a link would embarrass you if a customer read it, it is also the description an agent would use to decide the page is not worth fetching. Writing those lines well is the only part of this exercise whose value does not depend on anyone adopting the proposal.

The guide closes with the decision itself: what to allow and what to block.

[^llmstxt-definition]: llmstxt.org, retrieved 2026-09-03. [^llmstxt-problem]: llmstxt.org, retrieved 2026-09-03. [^llmstxt-format]: llmstxt.org, retrieved 2026-09-03. [^llmstxt-optional]: llmstxt.org, retrieved 2026-09-03. [^mistral-llmstxt-live]: docs.mistral.ai/llms.txt, retrieved 2026-09-03.