X-Robots-Tag

Robots.txt decides whether a crawler may fetch a URL. Once it has fetched it, a second set of rules decides what a search engine may do with it: index it or not, show a snippet or not, how long a snippet. Those rules can sit in the page's HTML, as a robots meta tag, or in the HTTP response, as an X-Robots-Tag header. The header is the one that works for everything.

How does the X-Robots-Tag header work?

The server sends it with the response, like any other header:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nosnippet

The rules are the same as the meta tag's. Google documents all, noindex, nofollow, none, nosnippet, indexifembedded, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex and unavailable_after. A rule can name one crawler, as in X-Robots-Tag: googlebot: nofollow; without a name it applies to every crawler that reads it.

When should you use the header instead of the meta tag?

A common case today is the plain-text copy. Many sites, ours included, publish a Markdown twin of each page for machines. A Markdown file has nowhere to put a meta tag, so any rule for it has to travel in a header: either X-Robots-Tag: noindex, which drops the twin from search entirely, or a Link: rel="canonical" header, which Google accepts for non-HTML files and which points search at the HTML page while leaving the twin readable.

Which rules reach AI answers?

For Google, the answer is documented. Google says nosnippet applies to every form of its search results, including AI Overviews and AI Mode, and also stops the content being used as a direct input for them. Its AI features page lists nosnippet, data-nosnippet, max-snippet and noindex as the controls for how content appears there.

That makes them blunt instruments. nosnippet keeps a page out of AI Overviews, and also removes its snippet from ordinary results. There is no Google rule that means "show me in search but not in AI answers".

Other AI engines do not document reading X-Robots-Tag for their answers. For ChatGPT, Claude and Perplexity the documented control is robots.txt, per crawler. Treat the header as a search-engine control, and robots.txt as the one every AI operator names.

What does it look like on a real site?

Here are the headers the Markdown twin of this site's RAG glossary page returned when we fetched https://attensira.com/glossary/rag.md on 4 October 2026:

HTTP/2 200
content-type: text/markdown; charset=utf-8
link: <https://attensira.com/glossary/rag>; rel="canonical", <https://attensira.com/license.xml>; rel="license"; type="application/rsl+xml"

There is no X-Robots-Tag on it, and that is a choice. A noindex header would keep the twin out of search, but it is the stricter rule, and the HTML page already carries the same words. The canonical header does the narrower job: it tells search engines which URL is the real one, so the HTML page is the one that ranks, and AI agents that ask for Markdown still get a clean copy. The HTML page sends no X-Robots-Tag either, so its rules live in the meta tag.

Check yours the same way: request a non-HTML URL with curl -I and read what comes back. If there is neither a noindex nor a canonical header, search engines decide on their own which copy to show.

What do people get wrong about it?

• Blocking the URL in robots.txt as well. Google says rules on a page that is disallowed from crawling are never found. If you want noindex to work, the crawler must be allowed to fetch the page.
• Sending it on the wrong responses. A header set for a whole path can catch pages you meant to keep. Check the response headers of a sample of URLs, not the config.
• Using nosnippet to hide from AI and losing search snippets. It does both at once.
• Assuming every AI engine reads it. Only some operators document it.
• Conflicting rules. If the header and the meta tag disagree, Google applies the more restrictive one.

How does Attensira help?

The free AI crawler URL inspector fetches a URL the way an AI crawler would and reports what blocks it. In a workspace, Attensira re-reads your pages and records a new version whenever the robots meta tag, the canonical link or the structured data types change, so a change that removed a page from answers can be traced to the day it happened. The robots rule it records is the meta tag, so check an X-Robots-Tag header with curl -I as above. When a rule is the cause, the crew's web engineer proposes the fix as a pull request for your approval.