X-Robots-Tag
The HTTP response header that carries indexing rules for any file type, how it differs from the robots meta tag, and which rules reach AI Overviews.
Karl-Gustav Kallasmaa, Founder & CEOLast updated The X-Robots-Tag is an HTTP response header that carries indexing and serving rules for a URL. It does the same job as the robots meta tag, from a place a meta tag cannot reach.
Google's specification states the equivalence directly: any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag. The simplest form is one line in the response:
X-Robots-Tag: noindexWhy the header exists
A meta tag lives in the <head> of an HTML document. A PDF, a video file, an image and a JSON payload have no head. Google's guidance is therefore explicit: to block indexing of non-HTML resources such as PDF files, video files or image files, use the X-Robots-Tag response header instead.
There is a second, more practical reason. The header is set in server or CDN configuration, so it can be applied to a whole path pattern at once — every file under /private/, every generated export — without editing any document. On sites where the documents are produced by a build step, that is the only place the rule can reliably live.
Multiple headers may be combined in one response, so a resource can carry, for example, a noimageindex rule and an unavailable_after rule at the same time.
Naming a crawler
The header may optionally specify a user agent before the rules:
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollowRules specified without a user agent are valid for all crawlers. The meta tag has a narrower equivalent: the name attribute takes the crawler's user agent token, and Google supports only two of them there — googlebot for all text results and googlebot-news for news results — with other values ignored. Both attributes are case-insensitive, as are the rule values.
Google also notes that it does not enforce placement of the robots meta tag in the head and will respect one found in the body of an HTML document. That is a tolerance, not a recommendation.
The rules worth knowing
The full list is in the specification, and Google publishes it in a machine-readable form too. The ones that decide behaviour most often:
- `noindex` — do not show this page, media or resource in search results.
- `nofollow` — do not follow the links on this page.
- `none` — equivalent to
noindex, nofollow. - `nosnippet` — do not show a text snippet or video preview for this page. A static image thumbnail may still appear where it makes for a better experience.
- `indexifembedded` — allow indexing of the content when it is embedded in another page through an iframe or similar, in spite of a
noindexrule. It only has an effect alongsidenoindex. - `max-snippet: [number]` — use at most that many characters as a textual snippet.
0is equivalent tonosnippet;-1lets Google choose. - `all` — the default, with no effect when listed explicitly.
Google adds a caveat that applies to every one of them: it is possible that these rules may not be treated the same by all other search engines. A directive is a statement to one operator that another operator is free to interpret differently.
The part that matters for AI answers
Two of these rules are not only about the blue-link snippet any more.
nosnippet applies to all forms of search results — web search, Google Images, Discover, AI Overviews and AI Mode — and, in Google's own words, will also prevent the content from being used as a direct input for AI Overviews and AI Mode. max-snippet likewise limits how much of the content may be used as a direct input for them. Google notes an exception: the limit does not apply where a publisher has separately granted permission, for instance by supplying content as in-page structured data or under a licence agreement.
The practical consequence is that a legacy nosnippet added years ago to stop a competitor scraping a definition is, today, an instruction not to be used in an AI answer. It is worth auditing which of your URLs still carry one, and why.
For finer control there is data-nosnippet, an HTML attribute applied to elements within a page for text-level settings, which excludes specific sections rather than the whole document.
The precondition everyone forgets
None of this works on a page that is disallowed. Google states that these settings can be read and followed only if crawlers are allowed to access the pages that include them.
Blocking a URL in robots.txt and adding noindex to it is therefore self-defeating: the crawler never fetches the response, so it never sees the rule. If you want a page kept out of an index, it has to be crawlable enough to be told so.
Terms related to X-Robots-Tag
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
The two separate stages that decide whether a page can be retrieved at all, and the reason a serving rule on a blocked page is never read.
A machine-readable list of the URLs you want discovered, bounded by the protocol at 50,000 URLs and 50MB per file, and a hint rather than an instruction.
Google's AI-generated summary at the top of a results page, and the snippet controls that decide whether your page can appear inside one.