XML sitemaps

A sitemap is a list of the URLs a site wants crawlers to know about. Search engines find most pages by following links; a sitemap covers the ones links miss and tells crawlers when something changed. It is one of the few things in SEO that is fully specified, and also one of the most commonly faked.

How does an XML sitemap work?

The format comes from sitemaps.org. Each URL is a <url> with a <loc> and, optionally, a <lastmod>:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://attensira.com/glossary/robots-txt</loc>
    <lastmod>2026-10-04</lastmod>
  </url>
</urlset>

One file holds at most 50,000 URLs or 50 MB uncompressed. Larger sites split into several files and publish a sitemap index that lists them. Crawlers learn where the sitemap is from a Sitemap: line in robots.txt, or from a submission in Google Search Console or Bing Webmaster Tools.

What does Google actually use?

Google is explicit about it:

And the whole file is "merely a hint". Google may not download it, and listing a URL does not get it crawled or indexed.

The lesson is about lastmod. A sitemap generator that stamps every URL with the time of the last build makes every page look changed every day. A crawler that learns the dates are noise stops using them, for the whole site.

What does a real sitemap look like?

Here is how attensira.com builds its own. The robots.txt ends with Sitemap: https://attensira.com/sitemap-index.xml, and that index lists twelve files, /sitemap/0.xml to /sitemap/11.xml, one or more per content family: static pages, compare, features, tools, glossary, blog, guides, bots and the rest.

The dates come from git, not from the build clock. A script finds the last commit that touched the files that actually produce each page and writes that date. A page whose files have no history gets no lastmod at all, rather than a made-up one. A change to a shared template dates the family's index page, not every page under it, so one template fix does not re-stamp hundreds of URLs on the same day. There is no changefreq and no priority, because nobody reads them.

Pages from the content tree, like this one, are dated by their own last-updated field. This page's lastmod changes when its content changes, and at no other time.

Do sitemaps matter for AI search?

Indirectly. AI search engines find pages through their own crawlers and through search indexes such as Bing's and Google's, and those use sitemaps for discovery like any crawler. No AI operator documents a sitemap as a reason to cite a page. A sitemap gets a page found sooner; whether the answer uses it depends on what the page says.

Attensira reads sitemaps for a different reason. It keeps an inventory of every URL your site publishes, so the crew can reason about pages no engine has cited yet. A web reader maps the domain; when it is unavailable, the inventory falls back to /sitemap.xml or the sitemaps your robots.txt declares, and then a page missing from your sitemap is missing from the inventory too.

What do people get wrong about sitemaps?

• Listing URLs that are not canonical. Redirects, parameters and noindex pages in a sitemap send mixed signals. List only the URL you want indexed.
• Fake lastmod dates. Stamping the build time on everything teaches crawlers to ignore the field.
• Forgetting the robots.txt line. A sitemap no crawler knows about does nothing.
• Blocking the listed URLs in robots.txt. A URL can be in the sitemap and disallowed at the same time; the disallow wins.
• Treating submission as indexing. It is a request to look, not a promise to keep.

How can you check yours?

The free XML sitemap validator fetches a sitemap or index and checks it against the protocol: the namespace, the URL and size limits, lastmod formats, duplicates, URLs on another host, and whether robots.txt declares it. In a workspace, sitemaps at the root are among the files the crew's web engineer may change, and a fix arrives as a pull request like the one at the top of this page, waiting for your approval. That is the service of an SEO agency at the price of a tool.