Choosing which AI crawlers to allow or block

What did we decide for our own site?

The table at the top of this page is the decision as it stands in attensira.com's robots.txt on 4 October 2026, grouped by the job each agent does. Here is what each choice costs or keeps for us.

• AI search indexing, allowed: keeps us eligible for those products' answers.
• Classic search, allowed: keeps us in Google and Bing, and in the AI features built on them.
• Live fetches for a person, allowed: lets assistants read our pages when someone asks about us.
• Training with an operator opt-out, allowed: our pages may be used to train future models.
• Meta-ExternalAgent, allowed: the heaviest load on our server by far (see the load figures further down).
• Bytespider, allowed: thousands of requests in the month, for a purpose no published documentation states; neither engine we measure, ChatGPT or Perplexity, is run by its operator, ByteDance.
• CCBot, blocked: our pages stay out of Common Crawl's open dataset.
• AhrefsBot and SemrushBot, blocked: our pages stay out of those tools' indexes.
• /api/, blocked for everyone: nothing a crawler should read lives there.

Every row is a trade, and ours is one defensible set among several. A company whose aim is to be known by AI systems has little reason to opt out of training. A publisher that sells its writing has every reason to. Both should allow the search agents, unless they have decided they do not want to appear in AI answers at all.

What we would not defend is a single rule for "AI crawlers". The table at the top has eight jobs because the agents do eight different things.

What should you allow if you want AI visibility?

Every agent that feeds today's answers.

Search agents. OAI-SearchBot surfaces sites in ChatGPT's search features. Claude-SearchBot improves Claude's search results. PerplexityBot surfaces and links sites in Perplexity, and Perplexity recommends allowing it. Googlebot feeds Google Search and, through its index, AI Overviews and AI Mode. bingbot feeds Bing. Block any of these and you remove yourself from that product's pool of sources.

User agents. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person's question needs it. Anthropic says blocking Claude-User may reduce your visibility in user-directed search. These visits are the closest thing to a person asking about you right now.

There is no documented upside to blocking either group for a site that wants to be found, and a clear cost.

How should you decide on training agents?

As a question about your content, not your visibility.

By the operators' own descriptions, blocking a training agent does not remove you from today's AI answers. GPTBot is OpenAI's training agent; ChatGPT search uses OAI-SearchBot. ClaudeBot collects training data; Claude's search and fetches use other agents. Google says Google-Extended does not affect inclusion or ranking in Search. Apple says pages that disallow Applebot-Extended can still appear in its search.

So the trade is about the future: whether models trained later may learn from your pages. Three questions settle it for most sites.

1. Is your content the product? If you sell articles, data, courses or research, training use is a rights question, and blocking is a reasonable default.
2. Do you want models to know your name without being prompted? A company that wants to be recommended has a reason to be in training data, though no operator promises that inclusion helps.
3. Can your servers take the load? Training crawlers can be heavy. On our site, Meta's agent alone made about nine in ten of all AI crawler requests.

For mixed-job tokens, where one agent does training and indexing (Meta-ExternalAgent, Amazonbot, Applebot), the decision is all or nothing for that token. Where the operator offers a separate opt-out, Google-Extended and Applebot-Extended, use it rather than blocking the main crawler.

What should different kinds of site do?

Three common starting points, each a robots.txt you can adapt.

A company that wants to be recommended, such as a SaaS product, an agency or a consultancy. Allow everything that answers questions; decide training on your own terms.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /account/

User-agent: *
Allow: /
Disallow: /account/

A publisher that sells its content. Opt out of training explicitly, keep search access if referral traffic matters, and put paid content behind authentication.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /subscribers/

A site with private or regulated sections. Same as either of the above, plus authentication on anything private. robots.txt is not access control, and some user-triggered fetchers are documented as ignoring it.

What about agents that do not follow robots.txt?

Some are documented as not always following it, and that changes what blocking can achieve.

OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt. Meta says Meta-ExternalFetcher may bypass it. In each case the operator's reasoning is the same: a person asked for the page, so the fetch behaves like a browser. Anthropic, by contrast, says its bots honour robots.txt, Claude-User included.

For a site that wants visibility, none of this is a problem; these are the fetches you want. For a site that wants to keep content away from AI products, it means robots.txt is not enough. Content that must not be read belongs behind a login, or behind server rules that refuse the request outright. And agents from operators that publish no documentation, such as Bytespider on our own site, give you nothing to rely on either way; treat them as unknown clients.

What about server load?

It is a real reason to act, and a separate one from visibility or rights.

On attensira.com, Meta-ExternalAgent alone made 108,487 requests in the 30 days to 4 October 2026, roughly fifteen times as many as all of OpenAI's agents together. For a small site on modest hosting, a crawler at that rate can be a cost and a reliability risk.

Three options, from gentlest to bluntest:

• Crawl-delay in that agent's group. Some operators honour it, Anthropic among them; Amazon says Amazonbot does not; it is not part of the robots.txt standard.
• A rate limit at your server or CDN for that agent, verified by IP. Predictable, and it does not depend on the crawler's cooperation.
• A disallow for that agent. Decide by its job: blocking a training-or-indexing agent such as Meta's costs nothing in the answer engines we track, while rate-limiting a search agent can cost citations.

Whatever you choose, make the load decision per agent. Throttling every crawler because one is heavy punishes the agents that feed your visibility for the behaviour of one that does not.

What about crawlers that are not AI at all?

Your robots.txt also governs agents that have nothing to do with AI answers: SEO tools' crawlers, link-preview fetchers, archive crawlers. Ours blocks two SEO tools' crawlers, AhrefsBot and SemrushBot, outright. That is a decision about which third-party indexes hold a copy of our pages, and it has no documented effect on AI answers in either direction.

Keep those decisions in clearly labelled groups of their own, so nobody later mistakes them for an AI policy, and so a change to one does not quietly change the other.

Which mistakes are expensive?

Blocking everything under the wildcard to stop training. It removes you from every AI search product that honours robots.txt.

Blocking the right job under the wrong token. A rule for OAI-SearchBot written to stop training removes you from ChatGPT search and leaves training untouched.

Blocking Googlebot to stop Gemini training. Google-Extended is the control for that. Blocking Googlebot removes you from Google Search.

Assuming robots.txt protects private content. It is a request to compliant crawlers. User-triggered fetchers such as Perplexity-User and Meta-ExternalFetcher are documented as able to ignore it.

Writing the file once. Operators add agents. Mistral now documents three, where our September 2026 notes on its page listed one. Each new token you have not named gets your wildcard rules.

What should the finished file say about each job?

Before shipping, read your robots.txt and write one line per job: search agents, user agents, training agents with opt-out tokens, mixed-job agents, open corpora, non-AI crawlers. Next to each, the decision and the reason. If you cannot write the line, the file is not making a decision about that job; the wildcard is making it for you.

That one-line-per-job summary is also the most useful thing to keep in version control next to the file. The next person to touch robots.txt, which may be you in a year, will know what each group is for and what removing it would cost.

How do you check the decision is working?

Three checks, a week after any change.

1. Logs: the agents you allowed still fetch your key sections; the ones you blocked stopped, after the day or so operators say changes take.
2. Verification: the agents in your logs are who they claim to be. The verification chapter covers how.
3. Answers: the questions you care about still cite your pages on each engine. Blocking the wrong agent shows up here first, as citations that stop.

Access is only the start. Once the right agents can read your site, what decides whether you are named is what your pages say and who else says it, the subject of the AI search visibility guide.