Training crawlers versus search agents

Which crawler jobs did our own site serve last month?

Here are the requests to attensira.com in the 30 days to 4 October 2026, read over the Attensira MCP that morning, grouped by the job each operator says its agent does, as Attensira classifies them. Attensira counts them at the edge by user agent; they are evidence of coverage, not a verified log.

• Training, or training and indexing mixed

  Agents that made requests

  Meta-ExternalAgent, GPTBot, ClaudeBot

  Requests

  most of about 120,000

  Feeds today's AI answers?

  No, or not separably

• Search indexing for an AI product

  Agents that made requests

  OAI-SearchBot, PerplexityBot

  Requests

  several thousand

  Feeds today's AI answers?

  Yes

• Classic search indexing

  Agents that made requests

  Googlebot, bingbot

  Requests

  697

  Feeds today's AI answers?

  Yes, for the engines built on them

• A live fetch for someone's question

  Agents that made requests

  ChatGPT-User, Claude-User

  Requests

  a few hundred at most

  Feeds today's AI answers?

  Yes, for that answer

• Undocumented: the operator publishes no description we could cite

  Agents that made requests

  Bytespider

  Requests

  3,205

  Feeds today's AI answers?

  Unknown

The detail behind the rows is where the lesson sits. Meta-ExternalAgent alone made 108,487 requests, about nine in every ten, and Meta describes its job as training models or improving products by indexing content: from outside, you cannot tell which. OpenAI's requests, at the top of this page, split three ways by user agent: mostly its search agent, then its training agent, then a few live fetches for a person's question.

So one operator sent three agents doing three jobs, and the busiest agent on the site does a job that feeds none of the answers we measure. A rule that treats "AI crawlers" as one thing gets at least one of those wrong.

What is the difference between training and retrieval?

Training is what a model learned before it was released. Retrieval is what it reads while answering.

Training turns a vast collection of pages into a model's general knowledge. A model trained on your pages may know your brand, your category and what you sell. It does not keep a link to the page it learned from, so when it mentions you from that knowledge, it cannot cite you.

Retrieval happens at answer time. The assistant searches an index, or fetches a page directly, reads passages and attaches some of them as sources. This is the only route to a citation. It is also the route to being described from your current pages rather than from whatever the model saw months ago.

The two are fed by different agents. Training crawlers feed the first. Search agents and user-triggered fetchers feed the second. That is why "should we block AI crawlers?" has no single answer: it is two questions.

Our own tracked question "what is GPTBot" shows retrieval at work. In the month, ChatGPT attached a link to attensira.com in 9 of its 30 answers. It named us in none. Our page was retrieved as background, which is a citation without a mention; the measurement guide covers that distinction.

What does blocking each job cost?

By each operator's own description, fetched on 4 October 2026:

• GPTBot

  Operator's description of the effect

  Content not used to make OpenAI's foundation models more useful and safe

  Effect on today's AI answers

  None stated; ChatGPT search uses OAI-SearchBot

• OAI-SearchBot

  Operator's description of the effect

  Site not surfaced in ChatGPT's search features

  Effect on today's AI answers

  You drop out of ChatGPT search answers

• ClaudeBot

  Operator's description of the effect

  Future materials excluded from Anthropic's training datasets

  Effect on today's AI answers

  None stated

• Claude-SearchBot or Claude-User

  Operator's description of the effect

  May reduce your visibility in user-directed search

  Effect on today's AI answers

  Fewer or no Claude citations

• PerplexityBot

  Operator's description of the effect

  Not surfaced and linked in Perplexity results

  Effect on today's AI answers

  You drop out of Perplexity's index

• Google-Extended (token)

  Operator's description of the effect

  Content not used for Gemini training and grounding

  Effect on today's AI answers

  None in Search, by Google's statement

• Applebot-Extended (token)

  Operator's description of the effect

  Content not used to train Apple's foundation models

  Effect on today's AI answers

  None; pages can still appear in Apple's search

• CCBot

  Operator's description of the effect

  Content left out of Common Crawl's open corpus

  Effect on today's AI answers

  None directly; the corpus feeds many future models

The shape is consistent. Blocking a training job costs you, at most, a say in what future models learn about you. Blocking a search or user job costs you citations now.

Which agents mix training and search?

Several, and for these one token covers both jobs, so you cannot split them with robots.txt.

• Applebot powers search in Spotlight, Siri and Safari, and Apple says its data may also help train Apple's models. Apple gives you the split another way: Applebot-Extended is a separate opt-out from training that does not crawl and does not affect search.
• Amazonbot is used to improve Amazon's products and may be used to train its AI models. Amazon documents two other agents, Amzn-SearchBot and Amzn-User, as not used for generative training.
• Meta-ExternalAgent crawls for training foundation models or for improving products by indexing content. Meta's user-triggered fetcher, Meta-ExternalFetcher, is separate and may bypass robots.txt.
• Googlebot is a search crawler, and Google offers the training split through the Google-Extended token instead.

Where an operator offers a separate training token, use it. Where it does not, the decision for that agent is all or nothing.

How can you tell which job a visit was for?

From the token, in most cases, and not at all in a few.

Where an operator runs one agent per job, the token tells you the job. A request from OAI-SearchBot is search indexing; one from GPTBot is training; one from ChatGPT-User is a fetch for a person. That is how the split of OpenAI's requests to our site above was made.

Where one agent does two jobs, the token cannot tell you which one a particular request served. Meta-ExternalAgent's 108,487 requests to our site could be training, indexing for Meta's products, or both. Nothing in the request says.

A third signal helps with user-triggered fetches: when a person follows a link from an assistant's answer to your site, the visit usually arrives from a browser with the assistant as the referrer. Attensira's AI traffic view records both kinds of visit, the agent's fetch and the person's click, and keeps them apart, because one says an assistant read the page and the other says a person chose to come.

Two practical rules follow. Make robots.txt decisions per token, using the operator's description of that token. And when you report crawler activity, report it by job, with the mixed-job agents in their own row rather than forced into one column.

What happens between a search crawl and a citation?

Several steps, and the crawler only controls the first. A search agent fetches the page and adds it to an index. When a person asks a question, the assistant searches that index, often several times for one question, picks passages from the pages it finds, and attaches some of those pages as sources. A page can be indexed and never retrieved for any question you care about, or retrieved and used only as background.

So allowing search agents is necessary for citation and nowhere near sufficient. The AI search visibility guide follows a page through each step.

Do user-triggered fetchers follow robots.txt?

Not all of them, and the operators say so.

• OpenAI says robots.txt rules may not apply to ChatGPT-User, because its requests follow user actions.
• Perplexity says Perplexity-User generally ignores robots.txt, because a user requested the fetch.
• Meta says Meta-ExternalFetcher may bypass robots.txt.
• Anthropic says its bots, including Claude-User, honour robots.txt.

The reasoning is that a person who pastes a link or asks about a page is acting like a browser. For most sites this is the visit you want most: someone is asking an assistant about you right now. If you truly need to keep a page away from these fetches, put it behind authentication on the server; robots.txt is not access control.

What about content already collected?

robots.txt applies to future crawls. A rule you add today does not remove pages already in a training set, and does not change models already trained on them. Blocking training crawlers is a decision about what future models learn from your future pages.

That cuts both ways. There is no rush: a decision made next month loses you very little. And there is no undo for the past: if keeping content out of training matters to you, the earlier you decide, the more of it stays out.

How should you decide?

Separate the two questions and answer each on its own terms.

For search and user agents: if you want to be named and cited in AI answers, allow them. There is no documented benefit to blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or bingbot for a site that wants visibility, and a clear cost.

For training agents: this is a business and rights decision, not a visibility one. Publishers who sell their content have reasons to block. A company that wants models to know its name has reasons to allow. Either way, by the operators' own descriptions, it does not change whether today's answers can cite you.

Our own choice is visible in our robots.txt: we allow the training agents of OpenAI, Anthropic and Meta, and the search and user agents of every engine we track, and we disallow Common Crawl's CCBot. The what to allow and what to block chapter works through the decision for different kinds of site, and the robots.txt chapter shows how to write it.