# Training data optimization

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> Training data optimization means publishing so future AI models learn the right facts about you. Which crawlers control it, why it is slow, and how to tell.

- Canonical page: https://attensira.com/glossary/training-data-optimization
- Family: glossary
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=training-data-optimization

**In short:** Training data optimization is publishing to shape what future AI models learn about you, so they describe you correctly when they answer without searching. You control access through each operator's training crawler in robots.txt. It is the slowest lever in AI search: a change shows up only after a new model is trained and released.

| Figure | What it counts | n |
| --- | --- | --- |
| 93.7% | of AI-crawler requests were Meta-ExternalAgent or Bytespider | 134,610 of 143,601; both classed training in our traffic log |
| 11% | of OpenAI's requests were classed as training (GPTBot) | n=7,536 OpenAI requests |
| 86% | of OpenAI's requests were classed as search indexing (OAI-SearchBot) | n=7,536 OpenAI requests |

Window 2026-09-05 to 2026-10-04. attensira.com AI traffic, by the user agent each request claimed, Attensira MCP.

_Source: attensira.com AI traffic, read over the Attensira MCP, as of 2026-10-04 (n=143,601 requests)._

## Key facts

- Also known as: Optimizing for model training; pre-training visibility
- What it targets: What a model says without searching
- Controlled by: [Each operator's training crawler, in robots.txt](/glossary/robots-txt)
- Time to effect: A model release: months, and never dated per page
- Related term: [Knowledge cutoff: the newest fact a model can know](/glossary/knowledge-cutoff)
- In Attensira: [Crawler fetches counted at the edge, not training](https://docs.attensira.com/traffic/ai-traffic)

An AI assistant answers in one of two ways. It searches the web and writes from what it finds, or it answers from what it learned in training. Most of what gets written about AI visibility concerns the first. Training data optimization concerns the second: making sure that when the next model is trained, the pages it learns from say the right things about you.

## How does a page get into training data?

Through a crawler whose job is collection for training. Each operator names its own, and documents it separately from the crawlers that fetch pages for search answers:

| Operator | Training crawler or token | Search or answer crawlers |
| --- | --- | --- |
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Google | Google-Extended (robots.txt token only) | Googlebot |
| Meta | Meta-ExternalAgent | Meta-ExternalFetcher |
| Common Crawl | CCBot | none |

Common Crawl matters out of proportion to its size because its public archive is a common ingredient in model training. Blocking it opts you out of every model that uses that archive, not just one.

The split is what makes a training decision independent of a search decision. OpenAI says disallowing GPTBot asks it not to use your content for training; ChatGPT search uses OAI-SearchBot, a different rule.

## Does an AI engine keep training and search apart?

ChatGPT does when asked. In our own workspace on 4 October 2026 it answered 'what is GPTBot' with the same split this page draws, citing OpenAI's bots page. Our /bots/gptbot page was among the pages it consulted, but the answer did not name Attensira.

**ChatGPT, asked 2026-10-04:** "what is GPTBot"

> GPTBot is OpenAI’s web crawler (a type of automated program that visits websites) used to collect publicly available web content that may be used to improve and train OpenAI’s generative AI foundation models. GPTBot ≠ ChatGPT Search crawler: GPTBot is for potential training data collection. OpenAI uses a separate crawler, OAI-SearchBot, for helping surface websites in ChatGPT search features.

Sources cited, in order: developers.openai.com.

Attensira: not named.

_Source: attensira.com workspace, read over the Attensira MCP, as of 2026-10-04 (1 stored answer, ChatGPT, 2026-10-04)._

## What do the training crawlers actually do on a real site?

This is attensira.com, which allows every training crawler except CCBot. Counted by Attensira at the edge in the 30 days to 4 October 2026, out of 143,601 AI crawler requests, each labelled by the user agent it claimed:

- **Meta-ExternalAgent made 131,105 requests across 873 pages,** about nine in ten of the total. Meta documents it as crawling for "training foundation AI models or improving products by indexing content directly", so one token covers both uses; our traffic log classes it as training. Nearly all of them (130,718) came in the last seven days, so this is a burst, not a weekly rate.

- **Bytespider made 3,505.** Our traffic log classes it as training, but ByteDance publishes no documentation for it that we could cite, so that label is ours, not ByteDance's.

- **OpenAI made 7,536**, and only about 11% of those were classed as training, which is GPTBot's job. Most, about 86%, were search indexing by OAI-SearchBot, which feeds ChatGPT search.

Volume says nothing about value. The crawler that fetched the most pages gives us no measurable answer today, and we cannot know whether any of those pages reached a model. We allow it anyway, because a future Meta model that describes Attensira from our own pages is better than one that describes us from other people's.

Across the web, sites are more willing to block training than search. In [Attensira's study of the top 10,000 Tranco domains](/data/ai-crawler-robots-access), run on 3 September 2026 over 8,036 readable robots.txt files, 12.4% blocked Meta-ExternalAgent at the root and 12.3% blocked the Google-Extended token, against 5.8% for OAI-SearchBot.

## Why is training the slowest lever?

Because three clocks must all run out. The crawler must collect the page. The operator must train a new model on a dataset that includes it. Then that model must be released and used. Operators publish knowledge cutoffs, which bound the newest fact a model can know without searching, but not when they collected any given page or whether it made it into a training set.

Compare retrieval. OpenAI says a robots.txt change reaches its search systems in about 24 hours, and a page that can be fetched can be cited in an answer the same week. Anything time-sensitive, such as a price, a launch or a correction, belongs on the retrieval path.

## What genuinely influences what a model learns?

- **Being collectable.** Allow the training crawlers you want, serve real HTML, and do not hide facts behind scripts or logins.

- **Saying one thing everywhere.** A model learns the description that recurs. Your own pages, your documentation and the third-party pages about you should agree on what you are and what you cost.

- **Stating facts plainly.** "Attensira is an AI-native SEO agency" in a first sentence is learnable. The same fact implied across three paragraphs is not.

## What do people get wrong about it?

- **Blocking everything named after an AI company.** A rule written to stop training can remove you from AI search answers if it also catches the search crawler.

- **Expecting a model to forget.** Blocking now does not remove what earlier models learned.

- **Measuring it with one chat.** One answer from one model is one observation. Answers vary run to run.

- **Treating it as separate from retrieval.** What a model already believes about you shapes how it reads the pages it fetches.

## How does Attensira handle it?

Attensira does not claim to measure training. It reads what each engine answers about your category every day, which reflects training and retrieval together, and it counts which crawlers fetch your pages at the edge, where JavaScript-free crawler requests are visible. When the answers describe you wrongly from memory, the crew's work is to fix the pages engines learn from and read, your own and the third-party ones, and to keep your [robots.txt](/glossary/robots-txt) allowing the crawlers you chose. The [AI crawler guide](/guides/ai-crawler-guide) covers every operator's crawlers in detail.

## FAQ

### What is training data optimization?

The practice of making sure the pages future models learn from describe your brand accurately: allowing the training crawlers you want, publishing clear and consistent facts, and being described the same way on other sites. It targets what a model knows without searching, as opposed to what it finds at answer time.

### Which crawlers decide whether my content is used for training?

Each operator names its own. OpenAI's is GPTBot, Google's opt-out is the Google-Extended token, Meta's is Meta-ExternalAgent, Common Crawl's is CCBot, whose public archive many models are trained on. They are separate from the crawlers that fetch pages for AI search answers, so you can allow one and block the other.

### How long does training data optimization take to work?

As long as it takes for a model trained after your change to be released and used, which is months, and no operator publishes when your page was collected or whether it was used. Retrieval changes can show up in days; training changes cannot be timed.

### Should I block AI training crawlers?

It is a business decision, not an SEO one. Blocking keeps your text out of future training sets that honour robots.txt; allowing it lets future models learn your facts. Blocking a training crawler does not remove you from AI search answers, which use different crawlers.

### Can I measure whether a model was trained on my content?

Not directly. You can see training crawlers fetch your pages in your server logs, and you can ask models questions with search turned off. Neither proves a page was in a training set. Attensira measures what the engines answer, which reflects training and retrieval together.