What to allow and what to block

Which AI crawlers should I allow and which should I block?

Allow every documented search and user-triggered agent. Decide the training tokens deliberately, one operator at a time, against your own position on reuse. That split keeps every citation route open while making the training question an explicit choice — and it is what most sites intend when they say they want to control AI access.

The two questions, separated

The decision only becomes tractable once it stops being one question.

Question one: do you want to appear in AI answers? For almost every commercial site the answer is yes, and it is settled by what your business does rather than by anything technical. If yes, the search and user-triggered agents are allowed. There is no version of wanting citations that is compatible with blocking the agents whose documented purpose is surfacing and linking pages.[^perplexity-surface-and-link]

Question two: are you willing for your content to enter training corpora? This is a rights and business question with no technical answer at all. It has real arguments on both sides, it varies by industry, and it is the question this guide deliberately does not answer for you.

Because the operators publish separate tokens, the two answers are independent. That independence is the practical payoff of everything in the earlier chapters.[^openai-two-jobs]

What each configuration costs

Stated as tradeoffs, with the loss column filled in honestly.

Row four is not a strategy; it is the accidental result of a wildcard rule aimed at training. It appears here because it is common, not because anyone chooses it.

Row three deserves its caveat spelled out. It does not achieve total exclusion. OpenAI documents that robots.txt rules may not apply to ChatGPT-User,[^openai-user-not-bound] and Perplexity says the same of its user-triggered agent. Anthropic is the exception, documenting all three of its agents as honouring robots.txt.[^anthropic-honours-all] So blocking everything buys most of the reuse position and all of the citation cost.

By kind of site

Different businesses genuinely land in different places, and pretending otherwise would be the marketing version of this chapter.

Software and B2B services. Allow search and user agents. Your buyers ask assistants which tool to use, and an answer that names you is worth more than any theoretical control over training text about your product. The training decision is close to free either way, because your competitive advantage is not the prose on your marketing pages.

Documentation sites. Allow everything, including training. Documentation exists to be found and used, an assistant answering a question about your product correctly is a support cost avoided, and content in a training corpus improves the odds that a model describes your product accurately even without a citation.

Publishers and media. The genuinely contested case. Blocking training crawlers costs nothing measurable. Blocking search and user-triggered crawlers removes the citation, which is the only referral path an assistant offers — and whether an answer that cites you is worth more than the visit it may replace is a judgement about your revenue model, not a technical fact. This guide will not pretend to settle it.

Ecommerce. Allow search and user agents. Product and category pages are exactly what assistants retrieve for purchase-intent questions, and a blocked product page is a product page that cannot be recommended.

Sites with licensed or third-party content. Your obligations may already decide this. Where you do not hold the rights to permit reuse, blocking training crawlers is a compliance position rather than a marketing one, and it should be documented as such so the next person does not undo it.

The per-operator asymmetries that change the answer

Four cases where a general rule gives the wrong result.

Perplexity. There is no training crawler to block. PerplexityBot is stated as not used to crawl content for AI foundation models,[^perplexity-surface-and-link] so blocking it is pure citation loss with no reuse gain. Any AI blocklist that includes PerplexityBot for training reasons is a mistake.

Google-Extended. The lowest-cost restriction available. Google states it does not impact inclusion in Google Search and is not a ranking signal,[^google-extended-no-search-impact] which makes disallowing it a training opt-out with a documented absence of search consequences. If you want a training position and are nervous about side effects, this is the one to start with.

Applebot. The crawler you almost certainly want, because it powers Spotlight, Siri and Safari as well as feeding training.[^applebot-two-permissions] The training half is separable through Applebot-Extended. Blocking Applebot itself costs you Apple's search surfaces for no additional reuse benefit.

Amazonbot and Common Crawl. The two where separation is not available. Amazon documents one agent doing both jobs, product improvement and possible model training, without a second token.[^amazonbot-both-jobs] Common Crawl feeds an open repository whose downstream users are unbounded, though it does publish an opt-out registry alongside robots.txt.[^ccbot-open-repository] Both are all-or-nothing decisions, and both are the ones where a training objection has its clearest application.

A default worth arguing with

For a commercial site with no specific rights constraint, this is a defensible starting point rather than a recommendation dressed as a fact.

1. Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and

Applebot — the search surfaces.

1. Allow ChatGPT-User, Perplexity-User, Claude-User and MistralAI-User — the

highest-intent fetches you receive.

1. Decide GPTBot, ClaudeBot, Google-Extended, Applebot-Extended,

meta-externalagent, Amazonbot and CCBot as one deliberate group, and write the reason down next to the rule.

1. Disallow internal paths under the wildcard group regardless of the above.
2. Verify that the configuration is real, in logs, rather than assuming the

file is doing what it says.

Step three is the one worth spending an hour on, and step five is the one people skip. The directives are in the robots.txt chapter; the verification routine is in verifying a crawler.

Arguments for allowing training crawlers, stated fairly

The training block is the default position in a lot of commentary, so the case against it deserves to be made properly rather than waved at.

Corpus presence shapes how a model describes your category. A model that has read your documentation has a better chance of describing what your product does correctly, whether or not it links to you. There is no number attached to this, and anyone who offers one is guessing — but the mechanism is not implausible, and its absence is not measurable either.

Blocking is a growing maintenance burden. Every new operator is a new token you have to learn about and add. A block list is only as good as the last time someone reviewed it, and an out-of-date block list provides the feeling of a position without the substance.

The enforcement is voluntary. Every restriction discussed here works because an operator chose to honour it. That is a real commitment and worth respecting, but it is not a technical guarantee, and a policy that assumes it is one will be disappointed.

The alternative may be worse for you specifically. If your material is frequently summarised by others, being absent from corpora does not remove your ideas from models — it removes your version of them, and leaves the second-hand accounts.

None of these is decisive. They are here because a page that only made the case for blocking would be advocacy rather than a guide, and because the reader most likely to act on this chapter is the one weighing both.

Writing the decision down

Whatever you choose, record it where the next person will find it — a comment in robots.txt is enough, and it is the highest-value line in the file.

# AI crawler policy, reviewed 2026-09-03, owner: platform team.
# Search and user-triggered agents: allowed. We want citations.
# Training crawlers: disallowed pending a licensing position.
# Review with the quarterly robots.txt check; operators add tokens.

Three things that comment buys you. It stops a future teammate reverting the policy because they could not tell whether it was deliberate. It names an owner, which is what makes a quarterly review actually happen. And it dates the decision, so a reader can tell at a glance whether it predates whatever changed since.

What none of this controls

Three limits to hold on to, so the decision is not mistaken for more than it is.

It is prospective. Every directive here governs future fetches. Nothing in a robots.txt reaches content already in a corpus or an index.

It reaches only the operators you name. A token you did not list is a decision you did not make. That is the maintenance burden of any block-list approach, and the reason to review at the same quarterly cadence the operators change on.

Access is not visibility. Allowing every crawler makes citation possible and does not make it happen. Whether a page is retrieved and named depends on it being a good answer to a real question, which is a content problem that no robots.txt line has ever solved.

That last point is where this guide ends and the rest of the work begins. The crawler configuration is the precondition — cheap to get right, expensive to get wrong, and finished once it is correct. Start back at the pillar if you need the whole shape again, or at training versus search crawlers if the distinction is the part you came for.

[^google-extended-no-search-impact]: Google Search Central, common crawlers, retrieved 2026-09-03. [^perplexity-surface-and-link]: Perplexity crawler documentation, retrieved 2026-09-03. [^openai-two-jobs]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^openai-user-not-bound]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^anthropic-honours-all]: Anthropic support documentation, retrieved 2026-09-03. [^applebot-two-permissions]: Apple support, About Applebot, retrieved 2026-09-03. [^ccbot-open-repository]: Common Crawl, CCBot, retrieved 2026-09-03. [^amazonbot-both-jobs]: Amazon developer documentation, Amazonbot, retrieved 2026-09-03.