AI training data

How much of a site's AI traffic is training?

On attensira.com, nearly all of it, and almost all from one agent. The chart at the top is our crawler log for the 30 days to 4 October 2026, read over the Attensira MCP, keeping only the agents whose documented job is collecting training data. Every row is labelled by the user agent the request claimed; none was IP-verified.

Meta-ExternalAgent alone made 131,105 requests, 91% of everything AI crawlers sent us, across 873 pages. Read that number with care: 130,718 of those requests fell in the last 7 days of the window, so it is a burst, not a monthly habit. Bytespider made 3,505 requests across 122 pages. GPTBot, OpenAI's training crawler, made about 829: that is a derived figure, 11% of OpenAI's 7,536 requests, because the log reports OpenAI's agents together with a rounded purpose split.

Visits are not influence. A training crawl feeds a model that will be released later and frozen. A search crawl feeds answers people get this week. The agents behind this week's answers are on AI search, and they are a small fraction of the traffic. Anthropic's agents are on Anthropic: Anthropic documents ClaudeBot as its training crawler, while the log classes its requests to our site as search indexing, so read it as both.

What is AI training data?

It is the corpus a model learned from before release: web pages, books, code, licensed and generated data. Whatever the model "knows" without searching comes from it, compressed into the model's weights. It stops at the model's knowledge cutoff and changes only with the next training run.

That is why a model can describe your company from memory, with old facts, without fetching anything. And it is why the training path is slow to influence: something you publish today reaches a model's memory only if it is collected, kept and used in a future run.

How is training data different from retrieval?

• When it is read

  Training data

  Months before the model ships

  Retrieval at answer time

  Seconds before the answer

• Who collects it

  Training data

  Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider)

  Retrieval at answer time

  Search and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot)

• Can it be cited?

  Training data

  No link to attach

  Retrieval at answer time

  Yes, the fetched URL

• How fast your changes show

  Training data

  Next model release, if at all

  Retrieval at answer time

  Days, once re-fetched

Retrieval-augmented generation and grounding are the second column. For anything published after a model's cutoff, they are the only path in.

Should you block AI training crawlers?

It is a business decision, and it is separate from AI search. OpenAI documents GPTBot as crawling content that may be used in training, and OAI-SearchBot as the agent that surfaces websites in ChatGPT search. Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User. So you can refuse training and stay eligible to be cited, if you write the rules per agent in robots.txt.

Publishers have made that choice at scale. The Reuters Institute found that by the end of 2023, 48% of the most widely used news websites in ten countries blocked OpenAI's crawlers and 24% blocked Google's AI crawler, measured from archived robots.txt files.

For a brand that wants to be recommended, the usual reasoning runs the other way: being in training data is how a model comes to associate your name with your category without searching. Blocking training does not hurt today's answers. It may make tomorrow's memory of you thinner.

What are the objections about AI training crawlers?

"Blocking now removes us from existing models." It does not. A rule governs future fetches only.

"Training crawlers are the AI traffic that matters." They are the loudest, as our chart shows. Search agents are the ones behind this week's answers.

"We can optimise our way into the next model." You can make your facts consistent and widely published, which is what training data optimization means. You cannot control what a lab keeps.

"User agents tell us everything." Agent strings can be forged. Operators such as Anthropic and OpenAI publish IP ranges so a log entry can be verified.

How does Attensira help with training and retrieval?

Attensira's crawler logs show which AI agents fetch your pages, how often and for which job, so a robots.txt decision is made on evidence. Its crew keeps the facts engines repeat about you consistent across your site and the sources that cite you, and measures answers per engine to see what changed. Crawler logs shows a chart like the one above for your own site, and the crawler directory explains each agent.