The AI crawler guide · chapter 3
Robots.txt directives for AI crawlers
How do I write robots.txt rules for AI crawlers?
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I write robots.txt rules for AI crawlers?
Name each agent's documented token in its own group and state a rule for it. Because the tokens are separate per job, a robots.txt that expresses a training objection without a search objection has to enumerate the training tokens rather than rely on a wildcard. The blocks below do exactly that, and the parsing rules after them explain why the shape matters.
The four parsing rules that decide what your file does
Read these before the blocks. Most robots.txt files that behave surprisingly are correct against the author's mental model and wrong against the standard.
Specificity, not order, resolves a conflict. RFC 9309 states that the most specific match found must be used, and defines the most specific match as the one with the most octets.[^rfc9309-specificity] Allow: /blog/public/ beats Disallow: /blog/ for a URL under both, wherever the two lines sit in the file. Anyone carrying over first-match-wins intuition from an older parser will write rules that do the opposite of what they read.
Matching groups combine. If more than one group matches a user agent, RFC 9309 says the matching groups' rules must be combined into one group and parsed together.[^rfc9309-combine] Do not assume a named group cleanly supersedes the wildcard group; write the named group to be complete on its own.
Status codes flip the meaning. A 500-range response means a crawler must assume complete disallow. A 400-range response means it may access any resource on the server.[^rfc9309-status] A robots.txt behind a flaky proxy is a site-wide block during every incident, and a missing one is an open door — opposite results from two failures that look equally like breakage.
Size is capped. Crawlers must parse at least 500 kibibytes.[^rfc9309-size] Beyond that, a generated file enumerating thousands of paths may simply be truncated, and the truncation is silent.
Block training, allow citation
This is the configuration most site owners actually want when they say they want to block AI: stay out of training corpora, stay in the answers. Every token here is one the operator documents for itself.
# Training and bulk-corpus crawlers: disallowed.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Search and answer crawlers: allowed.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: MistralAI-User
Allow: /
User-agent: Applebot
Allow: /
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xmlThree notes on that block. Google-Extended and Applebot-Extended are permissions rather than crawlers — disallowing them changes how already-crawled content may be used, and Google states it does not affect Search inclusion or ranking.[^google-extended-token][^applebot-extended-token] The explicit Allow: / groups are not strictly required where nothing else would block those agents, but writing them documents the decision, which is worth more than the two lines cost. And the Sitemap line is host-level rather than group-level: it applies to the file, not to whichever group it happens to sit under.
Block everything AI, keep normal search
For sites that want no relationship with AI assistants but still want Google.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: CCBot
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: Googlebot
Allow: /Stacked User-agent lines before a single rule set are one group addressed to many agents — a legitimate and compact form. Be aware of what this block does not achieve: OpenAI states robots.txt rules may not apply to ChatGPT-User,[^openai-tokens-robots] and Perplexity says the same of Perplexity-User. Listing them is a clear statement of intent that some agents are documented as not bound by.
Allow everything
The default position for most commercial sites, written explicitly so the decision is recorded rather than inherited.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlAn empty or absent robots.txt has the same practical effect. Writing it out means the next person to touch the file knows the permissiveness was chosen.
Per-operator forms the operators themselves publish
Where an operator gives a canonical example, use theirs. Anthropic documents blocking with a ClaudeBot group and a Disallow: /, and documents Crawl-delay support across its three agents.[^anthropic-crawl-delay]
User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /private/Common Crawl documents the same shape for CCBot, and additionally operates an opt-out registry for sites that want exclusion from future crawls.[^ccbot-block-example]
User-agent: CCBot
Disallow: /Amazon's documentation is the one that changes what you can rely on: Amazonbot honours robots.txt, page-level meta tags including noindex and noarchive, and rel=nofollow, but does not support crawl-delay — and it may use a robots.txt cached within the previous 30 days.[^amazonbot-no-crawl-delay] Meta documents a shorter but similar caching window: up to 24 hours for a change to take effect.[^meta-cache]
That caching behaviour is the most common reason a correct new rule appears to be ignored. Before concluding a crawler is misbehaving, check how long the operator says its cache lives.
Which block do you want?
The middle row is the one to choose deliberately rather than drift into. It forgoes every AI citation route in exchange for control over reuse, and the operators that document their user-triggered agents as not bound by robots.txt mean it does not fully deliver even that.
Directives that exist, and directives people wish existed
The standard is small. RFC 9309 defines User-agent, Allow and Disallow, and that is the whole vocabulary a conforming crawler must understand. Sitemap is a widely honoured convention. Crawl-delay is an extension: Anthropic documents support for it across its three agents,[^anthropic-crawl-delay] Amazon documents that Amazonbot does not support it.[^amazonbot-no-crawl-delay] Support is per-operator and has to be read rather than assumed.
Everything else circulating as an AI-specific robots.txt directive is not part of the protocol. There is no standardised line that says "index but do not train", no field that grants a licence with conditions, and no directive that expresses a price. The training-versus-citation split is expressed by naming different tokens, which is precisely why the tokens are worth knowing exactly. A file containing an invented directive is not a stricter file; it is a file with a line every crawler ignores.
The one adjacent mechanism that is real is the page-level robots meta tag. Amazon documents Amazonbot honouring noindex and noarchive at the page level, and rel=nofollow at the link level.[^amazonbot-no-crawl-delay] That is a per-page control robots.txt cannot express, and it is worth remembering that a page-level tag can only be read if the crawler is allowed to fetch the page in the first place. Disallowing a URL in robots.txt prevents the crawler from ever seeing the noindex on it.
Testing a change
A robots.txt change is deployed code, and the same discipline applies.
- Fetch the file over the public internet, not from your development
machine. Confirm a 200 status and a text/plain content type. This single check catches the failure mode with the worst consequence, because a 500 here reads as a site-wide disallow.
- Read it as the crawler would. For each agent you care about, work out which
groups match it, combine them, and find the longest matching rule for a representative URL. If that exercise is hard, the file is too complicated.
- Wait out the caches before judging the result — up to 24 hours for Meta,
and potentially a 30-day-old cached copy for Amazonbot.
- Confirm in logs. The rule is not working because it is written correctly;
it is working when verified crawler traffic changes to match it. Attensira's crawler logs and any log pipeline you already run answer the same question.
Rules that are worth applying to every group
Four practices that survive contact with the tokens changing under you.
- Disallow your API and internal paths in the wildcard group. Nothing under
an /api/ prefix benefits from being crawled by anyone, and a permissive AI policy is not a reason to expose it.
- Never use robots.txt to hide something. It is a publicly readable list of
the paths you consider sensitive, and it does not prevent access.
- Keep the file small enough to read aloud. The 500-kibibyte parse floor is
generous; a robots.txt nobody can hold in their head is one whose behaviour nobody can predict.
- Re-check it at the same cadence you re-check the tokens. A file naming
agents that no longer exist, and missing ones that do, is a file that expresses last year's decision.
Two things robots.txt genuinely cannot do belong here as well, because people reach for it to do them. It cannot control what a user-triggered fetcher reads on a person's behalf where the operator says the rules may not apply, and it cannot reach anything sitting in front of your origin. A CDN bot ruleset that rejects a crawler after robots.txt has allowed it is a second, independent access decision made in a place robots.txt cannot see — and it is the most common cause of a crawler being blocked by a site that believes it is open.
If you would rather generate the file than hand-write it, the robots.txt generator produces these blocks, and the bot access score reports what your current file actually permits. Whichever route you take, verifying a crawler is the next step: a rule is only as good as your ability to tell whether the agent obeying it was the real one.
[^rfc9309-specificity]: RFC 9309, section 2.2.2, retrieved 2026-09-03. [^rfc9309-combine]: RFC 9309, section 2.2.1, retrieved 2026-09-03. [^rfc9309-status]: RFC 9309, sections 2.3.1.3 and 2.3.1.4, retrieved 2026-09-03. [^rfc9309-size]: RFC 9309, section 2.5, retrieved 2026-09-03. [^openai-tokens-robots]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^anthropic-crawl-delay]: Anthropic support documentation, retrieved 2026-09-03. [^amazonbot-no-crawl-delay]: Amazon developer documentation, Amazonbot, retrieved 2026-09-03. [^meta-cache]: Meta for Developers, web crawlers, retrieved 2026-09-03. [^ccbot-block-example]: Common Crawl, CCBot, retrieved 2026-09-03. [^google-extended-token]: Google Search Central, common crawlers, retrieved 2026-09-03. [^applebot-extended-token]: Apple support, About Applebot, retrieved 2026-09-03.
Questions people ask
- Where does robots.txt have to live?
- At the root of the host it governs, served as plain text — one file per scheme, host and port. A file at a subdirectory path is not read as robots.txt by conforming crawlers, and a rule for one subdomain does not govern another.
- Does the order of my rules matter?
- Not for allow and disallow within a matched group. RFC 9309 says the most specific match must be used, and defines specificity by the number of octets in the rule. A longer, more precise Allow beats a shorter Disallow no matter which was written first.
- What happens if my robots.txt returns an error?
- The two failure modes have opposite effects. RFC 9309 says a crawler must assume complete disallow when robots.txt is unreachable because of a 500-range status, and that a 400-range status means it may access any resource on the server. A broken robots.txt can silently block your whole site.
- How big can robots.txt be?
- RFC 9309 requires crawlers to parse at least 500 kibibytes, so anything beyond that may be ignored. That is generous for a hand-written file and a real constraint for a generated one that enumerates thousands of paths.
- Should I use Crawl-delay for AI crawlers?
- Only where the operator says it is supported. Anthropic documents Crawl-delay support for its three agents; Amazon documents that Amazonbot does not support it. It is an extension rather than part of the standard, so support is per-operator and has to be checked rather than assumed.
Sources
Every factual statement above, with the page it came from and the date that page was read.
RFC 9309 section 2.2.2 states that the most specific match found must be used, and defines the most specific match as the one that has the most octets.
rfc-editor.org · retrieved
“The most specific match found MUST be used. The most specific match is the match that has the most octets.”
RFC 9309 section 2.2.1 states that if more than one group matches the user agent, the matching groups' rules must be combined into one group and parsed together.
rfc-editor.org · retrieved
“If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group and parsed according to Section 2.2.2.”
RFC 9309 states that a crawler must assume complete disallow when robots.txt is unreachable because of a server error in the 500 range, and that a 400-range status means the crawler may access any resource on the server.
rfc-editor.org · retrieved
RFC 9309 section 2.5 states that the robots.txt parsing limit must be at least 500 kibibytes.
rfc-editor.org · retrieved
“The parsing limit MUST be at least 500 kibibytes”
OpenAI documents GPTBot, OAI-SearchBot and OAI-AdsBot as respecting robots.txt, and states that robots.txt rules may not apply to ChatGPT-User because its fetches follow user actions.
developers.openai.com · retrieved
Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot as honouring standard robots.txt directives and supporting the Crawl-delay extension, and gives a blocking example of a ClaudeBot group with Disallow slash.
support.claude.com · retrieved
Amazon documents that Amazonbot honours the Robots Exclusion Protocol, page-level robots meta tags including noindex and noarchive, and rel=nofollow, but does not support the crawl-delay directive, and may use a cached robots.txt from the previous 30 days.
developer.amazon.com · retrieved
Meta documents that changes to robots.txt can take up to 24 hours to take effect for its crawlers because of caching.
developers.facebook.com · retrieved
Common Crawl documents blocking CCBot with a robots.txt group naming CCBot and disallowing slash, and states that it also offers an opt-out registry for sites seeking exclusion from future crawls.
commoncrawl.org · retrieved
Google documents Google-Extended as a token controlling whether crawled content may be used for training Gemini models and grounding in Gemini Apps and Vertex AI, and states that it does not impact inclusion in Google Search nor act as a ranking signal.
developers.google.com · retrieved
Apple documents Applebot-Extended as the user agent through which publishers opt out of their website content being used to train Apple's general purpose foundation models, and states that Applebot-Extended does not crawl webpages.
support.apple.com · retrieved