XML sitemaps
A machine-readable list of the URLs you want discovered, bounded by the protocol at 50,000 URLs and 50MB per file, and a hint rather than an instruction.
Karl-Gustav Kallasmaa, Founder & CEOLast updated An XML sitemap is a file listing the URLs on a site that you want crawlers to know about, with optional metadata about each one. It is the cheapest discovery signal a site controls, and it is routinely asked to do jobs it cannot do.
What the protocol actually says
The Sitemaps protocol is short and worth reading once in full. Its load-bearing statements:
- Limits. Each sitemap file must have no more than 50,000 URLs and must be no larger than 50MB (52,428,800 bytes). A sitemap index may not list more than 50,000 sitemaps and is bound by the same size limit. Larger sites split across several files and publish an index.
- No guarantee. The protocol does not promise that a listed URL will be crawled or included. It is a hint about what exists, not a request that must be honoured.
- Hints are weak.
changefreqandpriorityare described as hints, and the protocol states that the priority you assign to a page is not likely to influence the position of your URLs in a search engine's result pages. - lastmod is the useful field. It tells a crawler what has actually changed since it last looked, which is the one question a sitemap can genuinely answer better than a crawl can.
That last point is where most of the value sits. A sitemap whose lastmod values are generated from real modification times is a change feed. A sitemap where every page claims to have changed today is noise, and a crawler learns to treat it as such.
How it differs from robots.txt
The two files are often edited in the same sitting and do opposite jobs.
- A sitemap says this exists and here is when it last changed. It is a discovery aid.
- Robots.txt says you may or may not fetch this. It is an access rule, standardised in RFC 9309.
Access wins. A URL listed in a sitemap and disallowed in robots.txt is not fetched; the listing does not override the rule, and publishing both is simply a contradiction a crawler resolves against you. The reverse mistake is subtler: removing a URL from the sitemap does nothing to remove it from an index, because a sitemap has no removal semantics at all. To take a page out you allow the crawl and serve noindex — see crawling and indexing and noindex vs nosnippet.
Does any of this matter for AI search?
A sitemap is an ordinary file served over HTTP, so any crawler may read it, including the AI agents that maintain their own indexes. No vendor commits to using it, and none of them treat it as a means of gaining access: OpenAI and Perplexity both document access through named user agents and robots.txt rather than through sitemaps. So the honest position is that a sitemap helps discovery on any crawler that chooses to consult it, and changes nothing about whether you are allowed to be fetched. The access decision is made in robots.txt, and the pool a retrieval system draws from is described in AI search.
There is one place a sitemap earns its keep specifically for AI surfaces. Google states that a page must be indexed and eligible to be shown with a snippet before it can appear as a supporting link in its AI experiences. Anything that shortens the path from publication to indexing therefore shortens the path to eligibility, and an accurate lastmod is the shortest lever you own.
How to act on it
- Generate it, never hand-maintain it. A stale sitemap is worse than none, because it teaches crawlers your signals are unreliable.
- Derive `lastmod` from real modification timestamps. If you cannot, omit the field rather than fake it.
- Split before you approach the limits, and publish a sitemap index. Do not wait to discover which URLs were dropped.
- Reference the sitemap from robots.txt so access rules and discovery live in one place.
- Include only canonical, indexable URLs. Listing pages you have blocked, redirected or marked
noindexis the most common way a sitemap loses its credibility. - Ignore `priority`. The protocol itself says it is unlikely to influence position; the time is better spent on the pages.
Frequently asked questions
How many URLs can one sitemap file hold?
No more than 50,000, and no larger than 50MB uncompressed.
Does listing a URL guarantee indexing?
No. The protocol does not guarantee that listed URLs will be crawled or included.
Are priority and changefreq worth setting?
Hardly. Accurate lastmod values are worth much more.
Do AI crawlers read sitemaps?
They can, but none guarantee it, and a sitemap never overrides an access rule.
Terms related to XML sitemaps
The two separate stages that decide whether a page can be retrieved at all, and the reason a serving rule on a blocked page is never read.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
The Schema.org vocabulary and the JSON-LD you embed with it, what it is genuinely good for, and the limits of what it can make happen.
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.