# AI training data

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> AI training data is the text a model learned from before release. How it differs from pages fetched at answer time, who collects it, with our own logs.

- Canonical page: https://attensira.com/glossary/ai-training-data
- Family: glossary
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=ai-training-data

**In short:** AI training data is the text, code and other material a model learned from before it was released. It shapes what the model knows without searching, frozen at its knowledge cutoff. It is collected by training crawlers such as GPTBot, ClaudeBot and Meta-ExternalAgent, which are separate from the search agents that fetch pages at answer time.

**Requests from training crawlers** (attensira.com, by the user agent each request claimed (none IP-verified); GPTBot derived as 11% of OpenAI's 7,536, 2026-09-05 to 2026-10-04, n=143,601 requests)

|  | Requests | Share |
| --- | --- | --- |
| Meta-ExternalAgent | 131,105 | 91.3% |
| Bytespider | 3,505 | 2.4% |
| GPTBot (about 11% of OpenAI's 7,536) | 829 | 0.6% |

_Source: attensira.com workspace, read over the Attensira MCP, as of 2026-10-04 (n=143,601 AI-crawler requests)._

## Key facts

- Also known as: Pre-training corpus
- Collected by: Training crawlers: GPTBot, ClaudeBot, Meta-ExternalAgent
- Frozen at: [The model's knowledge cutoff](/glossary/knowledge-cutoff)
- The other path: [Retrieval at answer time](/glossary/rag)
- Controlled in: [robots.txt, one rule per agent](/glossary/robots-txt)
- In the docs: [Traffic › AI traffic](https://docs.attensira.com/traffic/ai-traffic)

## How much of a site's AI traffic is training?

On attensira.com, nearly all of it, and almost all from one agent. The chart at the top is our crawler log for the 30 days to 4 October 2026, read over the Attensira MCP, keeping only the agents whose documented job is collecting training data. Every row is labelled by the user agent the request claimed; none was IP-verified.

Meta-ExternalAgent alone made 131,105 requests, 91% of everything AI crawlers sent us, across 873 pages. Read that number with care: 130,718 of those requests fell in the last 7 days of the window, so it is a burst, not a monthly habit. Bytespider made 3,505 requests across 122 pages. GPTBot, OpenAI's training crawler, made about 829: that is a derived figure, 11% of OpenAI's 7,536 requests, because the log reports OpenAI's agents together with a rounded purpose split.

Visits are not influence. A training crawl feeds a model that will be released later and frozen. A search crawl feeds answers people get this week. The agents behind this week's answers are on [AI search](/glossary/ai-search), and they are a small fraction of the traffic. Anthropic's agents are on [Anthropic](/glossary/anthropic): Anthropic documents ClaudeBot as its training crawler, while the log classes its requests to our site as search indexing, so read it as both.

## What is AI training data?

It is the corpus a model learned from before release: web pages, books, code, licensed and generated data. Whatever the model "knows" without searching comes from it, compressed into the model's weights. It stops at the model's [knowledge cutoff](/glossary/knowledge-cutoff) and changes only with the next training run.

That is why a model can describe your company from memory, with old facts, without fetching anything. And it is why the training path is slow to influence: something you publish today reaches a model's memory only if it is collected, kept and used in a future run.

## How is training data different from retrieval?

|  | Training data | Retrieval at answer time |
| --- | --- | --- |
| When it is read | Months before the model ships | Seconds before the answer |
| Who collects it | Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider) | Search and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) |
| Can it be cited? | No link to attach | Yes, the fetched URL |
| How fast your changes show | Next model release, if at all | Days, once re-fetched |

[Retrieval-augmented generation](/glossary/rag) and [grounding](/glossary/grounding) are the second column. For anything published after a model's cutoff, they are the only path in.

## Should you block AI training crawlers?

It is a business decision, and it is separate from AI search. OpenAI [documents](https://developers.openai.com/api/docs/bots) GPTBot as crawling content that may be used in training, and OAI-SearchBot as the agent that surfaces websites in ChatGPT search. Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User. So you can refuse training and stay eligible to be cited, if you write the rules per agent in [robots.txt](/glossary/robots-txt).

Publishers have made that choice at scale. The Reuters Institute found that by the end of 2023, 48% of the most widely used news websites in ten countries blocked OpenAI's crawlers and 24% blocked Google's AI crawler, measured from archived robots.txt files.

For a brand that wants to be recommended, the usual reasoning runs the other way: being in training data is how a model comes to associate your name with your category without searching. Blocking training does not hurt today's answers. It may make tomorrow's memory of you thinner.

## Does an AI answer tell training crawlers from search agents?

Not always. Asked for a list of AI crawler user agents on 4 October 2026, Perplexity listed GPTBot beside OAI-SearchBot, and training crawlers such as CCBot and Bytespider beside BingBot, with no word on which ones collect training data. A blocklist copied from it would not separate the two. Over 30 days, neither engine named or cited attensira.com in 60 answers to this question.

**Perplexity, asked 2026-10-04:** "list of AI crawler user agents"

> GPTBot · OAI-SearchBot · PerplexityBot · ClaudeBot · CCBot · Google-Extended · Applebot-Extended · Bytespider · Meta-WebIndexer · BingBot (and other major AI crawlers vary by source; see notes below). Notes: Exact lists vary across sources and updates in 2026; many pages consolidate 14–67 bots with current tokens and blocking guidance.

Sources cited, in order: pulserank.ai, technologychecker.io, quanticdata.io, contextbolt.com, anagram.ai, xseek.io, pleqo.com, geotoolbox.ai.

Attensira: not named.

_Source: attensira.com workspace, read over the Attensira MCP, as of 2026-10-04 (n=60 answers to this question in 30 days, Attensira named in 0)._

## What are the objections about AI training crawlers?

**"Blocking now removes us from existing models."** It does not. A rule governs future fetches only.

**"Training crawlers are the AI traffic that matters."** They are the loudest, as our chart shows. Search agents are the ones behind this week's answers.

**"We can optimise our way into the next model."** You can make your facts consistent and widely published, which is what [training data optimization](/glossary/training-data-optimization) means. You cannot control what a lab keeps.

**"User agents tell us everything."** Agent strings can be forged. Operators such as Anthropic and OpenAI publish IP ranges so a log entry can be verified.

## How does Attensira help with training and retrieval?

Attensira's crawler logs show which AI agents fetch your pages, how often and for which job, so a robots.txt decision is made on evidence. Its crew keeps the facts engines repeat about you consistent across your site and the sources that cite you, and measures answers per engine to see what changed. [Crawler logs](/features/crawler-logs) shows a chart like the one above for your own site, and the [crawler directory](/bots) explains each agent.

## FAQ

### What is AI training data?

It is the material, mostly text and code, that a language model learned from before release. It determines what the model knows without searching. That knowledge is frozen at the model's knowledge cutoff and changes only when the model is retrained.

### Is blocking the training crawler the same as blocking AI search?

No. OpenAI states that GPTBot crawls content that may be used in training and that OAI-SearchBot is the agent that surfaces websites in ChatGPT search. Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User. You can refuse training and stay eligible for answers.

### How many publishers block AI training crawlers?

Among large news sites, many. A Reuters Institute factsheet from February 2024 found that by the end of 2023, 48% of the most widely used news websites across ten countries blocked OpenAI's crawlers and 24% blocked Google's AI crawler, measured from archived robots.txt files.

### Does blocking a crawler remove my content from a model already trained?

No. A robots.txt rule governs future fetches. Material already collected and used in a finished training run is not withdrawn by a rule added afterwards, which is why the decision is worth making deliberately.

### Can a model quote my page if it was never in the training data?

Yes, through retrieval. An assistant with a search step fetches live pages at answer time and writes from them, so a page no training crawler collected can still be read, quoted and cited. For anything published after a model's cutoff, retrieval is the only way in.