Training data optimization

An AI assistant answers in one of two ways. It searches the web and writes from what it finds, or it answers from what it learned in training. Most of what gets written about AI visibility concerns the first. Training data optimization concerns the second: making sure that when the next model is trained, the pages it learns from say the right things about you.

How does a page get into training data?

Through a crawler whose job is collection for training. Each operator names its own, and documents it separately from the crawlers that fetch pages for search answers:

• OpenAI

  Training crawler or token

  GPTBot

  Search or answer crawlers

  OAI-SearchBot, ChatGPT-User

• Google

  Training crawler or token

  Google-Extended (robots.txt token only)

  Search or answer crawlers

  Googlebot

• Meta

  Training crawler or token

  Meta-ExternalAgent

  Search or answer crawlers

  Meta-ExternalFetcher

• Common Crawl

  Training crawler or token

  CCBot

  Search or answer crawlers

  none

Common Crawl matters out of proportion to its size because its public archive is a common ingredient in model training. Blocking it opts you out of every model that uses that archive, not just one.

The split is what makes a training decision independent of a search decision. OpenAI says disallowing GPTBot asks it not to use your content for training; ChatGPT search uses OAI-SearchBot, a different rule.

What do the training crawlers actually do on a real site?

This is attensira.com, which allows every training crawler except CCBot. Counted by Attensira at the edge in the 30 days to 4 October 2026, out of 143,601 AI crawler requests, each labelled by the user agent it claimed:

• Meta-ExternalAgent made 131,105 requests across 873 pages, about nine in ten of the total. Meta documents it as crawling for "training foundation AI models or improving products by indexing content directly", so one token covers both uses; our traffic log classes it as training. Nearly all of them (130,718) came in the last seven days, so this is a burst, not a weekly rate.
• Bytespider made 3,505. Our traffic log classes it as training, but ByteDance publishes no documentation for it that we could cite, so that label is ours, not ByteDance's.
• OpenAI made 7,536, and only about 11% of those were classed as training, which is GPTBot's job. Most, about 86%, were search indexing by OAI-SearchBot, which feeds ChatGPT search.

Volume says nothing about value. The crawler that fetched the most pages gives us no measurable answer today, and we cannot know whether any of those pages reached a model. We allow it anyway, because a future Meta model that describes Attensira from our own pages is better than one that describes us from other people's.

Across the web, sites are more willing to block training than search. In Attensira's study of the top 10,000 Tranco domains, run on 3 September 2026 over 8,036 readable robots.txt files, 12.4% blocked Meta-ExternalAgent at the root and 12.3% blocked the Google-Extended token, against 5.8% for OAI-SearchBot.

Why is training the slowest lever?

Because three clocks must all run out. The crawler must collect the page. The operator must train a new model on a dataset that includes it. Then that model must be released and used. Operators publish knowledge cutoffs, which bound the newest fact a model can know without searching, but not when they collected any given page or whether it made it into a training set.

Compare retrieval. OpenAI says a robots.txt change reaches its search systems in about 24 hours, and a page that can be fetched can be cited in an answer the same week. Anything time-sensitive, such as a price, a launch or a correction, belongs on the retrieval path.

What genuinely influences what a model learns?

• Being collectable. Allow the training crawlers you want, serve real HTML, and do not hide facts behind scripts or logins.
• Saying one thing everywhere. A model learns the description that recurs. Your own pages, your documentation and the third-party pages about you should agree on what you are and what you cost.
• Stating facts plainly. "Attensira is an AI-native SEO agency" in a first sentence is learnable. The same fact implied across three paragraphs is not.

What do people get wrong about it?

• Blocking everything named after an AI company. A rule written to stop training can remove you from AI search answers if it also catches the search crawler.
• Expecting a model to forget. Blocking now does not remove what earlier models learned.
• Measuring it with one chat. One answer from one model is one observation. Answers vary run to run.
• Treating it as separate from retrieval. What a model already believes about you shapes how it reads the pages it fetches.

How does Attensira handle it?

Attensira does not claim to measure training. It reads what each engine answers about your category every day, which reflects training and retrieval together, and it counts which crawlers fetch your pages at the edge, where JavaScript-free crawler requests are visible. When the answers describe you wrongly from memory, the crew's work is to fix the pages engines learn from and read, your own and the third-party ones, and to keep your robots.txt allowing the crawlers you chose. The AI crawler guide covers every operator's crawlers in detail.