AI training data
The corpus a model learns from, how it differs from what an assistant retrieves at query time, and how publishers have actually responded.
Karl-Gustav Kallasmaa, Founder & CEOLast updated AI training data is the corpus a model learns from before anyone asks it anything. It is fixed at training time, it is not a database the model can look things up in, and it is a different thing entirely from the documents an assistant fetches while answering. Conflating the two produces most of the bad decisions in this area.
Training is not retrieval
A model that has been trained on text carries that text only as adjusted weights. It can reproduce patterns and facts it absorbed, sometimes accurately and sometimes not, but there is no record it can point to. Retrieval is the opposite arrangement: the system fetches live documents at query time and the model writes from what it just read, which is why a retrieved answer can carry a working link and a purely trained answer cannot. Retrieval-augmented generation is that second architecture, and AI training vs AI retrieval sets the two side by side.
The consequences run in both directions and are worth stating plainly:
- A page never collected for training can still be quoted, today, through retrieval.
- A page that was in a training corpus is not removed from a finished model by a robots.txt rule added later.
- Refusing training therefore costs you nothing on the retrieval surfaces, provided you answer the two crawlers separately.
The operators publish the split
The separation is not something to infer; it is documented. OpenAI's crawler documentation describes GPTBot as used to crawl content that may be used in training its generative AI foundation models, and states that disallowing GPTBot indicates a site's content should not be used in training. It describes OAI-SearchBot separately as used to surface websites in search results in ChatGPT's search features. Perplexity's bot documentation states that PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for AI foundation models.
That is the whole reason a single blanket rule is a poor instrument. GPTBot vs OAI-SearchBot, CCBot vs GPTBot and Google-Extended vs Googlebot each work through one pair; robots.txt covers how the rules are expressed.
What publishers actually did
The one large public measurement of behaviour comes from the Reuters Institute for the Study of Journalism. Richard Fletcher's factsheet of 22 February 2024 tracked archived robots.txt files across 2023 for the 15 most used online news sources in Brazil, Denmark, Germany, India, Mexico, Norway, Poland, Spain, the UK and the US. By the end of 2023, 48% of those sites were blocking OpenAI's crawlers and 24% were blocking Google's AI crawler, and 97% of the sites blocking Google's were also blocking OpenAI's.
Three things in that are more interesting than the headline. The decision was nearly nested rather than independent, which suggests a single editorial policy applied across bots rather than a per-vendor calculation. The variation by country was enormous: blocking of OpenAI's crawlers ranged from 79% in the US to 20% in Mexico and Poland, and of Google's from 60% in Germany to 7% in Poland and Spain. And the population studied was news publishers, whose incentives are unusual — their product is the text itself. Nothing in the factsheet licenses a claim about how software vendors, retailers or professional firms behave.
How to act on it
- Read your logs first. Enumerate the agents actually fetching you before writing rules for agents that never visit.
- Answer training and retrieval as two separate questions, and write the reasoning down next to the rules. A future maintainer inherits the file without the argument.
- Do not expect retroactive effect. A rule added now governs future fetches only.
- Decide by what the content is for. If the page exists to be found by buyers, being retrievable is the point; if it is the product, the calculus is genuinely different.
- Revisit on a schedule. Agents and their published purposes change, and a robots.txt written two years ago is a policy nobody is currently defending.
Frequently asked questions
How many publishers block training crawlers?
Among the most used news sites in ten countries, 48% blocked OpenAI's crawlers by the end of 2023.
Is blocking training the same as blocking AI search?
No. The operators run separate agents so the decisions can differ.
Does a new rule remove me from an existing model?
No. It governs future fetches only.
Can I be quoted without ever being in the training data?
Yes, through retrieval at query time.
Terms related to AI training data
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
The units a language model actually reads and writes, why they are not words, and why every limit and price you meet is denominated in them.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.