Training data optimization
Publishing with the intent of shaping what a future model learns about you — why it is the slowest lever available, and which crawler decisions actually control it.
Karl-Gustav Kallasmaa, Founder & CEOLast updated Training data optimization is publishing with the deliberate intent that a future model, when it is trained, will have learned the correct things about your brand, product or topic. It operates on the model's weights rather than on any individual answer, which makes it the slowest and least verifiable lever in the whole field — and the reason it is worth understanding precisely rather than pursuing enthusiastically.
The mechanism, and its two clocks
A model's parametric knowledge is fixed at training time. Anthropic publishes both a training data cutoff and a reliable knowledge cutoff per model — May 2026 for Claude Opus 5, January 2026 for Claude Sonnet 5 — and describes the reliable cutoff as "the date through which the model's knowledge is most extensive and reliable". Anything published after that date does not exist to that model except by retrieval.
So there are two clocks, and they run at wildly different speeds:
- The retrieval clock ticks per query. A page edited this morning can be fetched and quoted this afternoon. See RAG.
- The training clock ticks per model generation. Content collected now competes for influence in a corpus assembled over months, then affects answers only once that model ships and only for as long as it serves.
Nothing you publish today reaches a model that has already been trained. This is the single fact that determines how much effort the discipline deserves.
Which agents control it
Training collection and answer-time fetching are done by different, separately addressable crawlers. Getting this distinction wrong is the most expensive mistake available here, because the two decisions look like one.
OpenAI documents GPTBot as the agent that crawls "content that may be used in training our generative AI foundation models", and states that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Separately, OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features", and ChatGPT-User covers "certain user actions in ChatGPT and Custom GPTs". OpenAI documents the two settings as independent — a publisher can allow one and disallow the other — while noting that if both are allowed it "may use the results from just one crawl for both use cases".
Google documents Google-Extended as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" — and, in the same sentence, for grounding, which Google glosses as "providing content from the Google Search index to the model at prompt time to improve factuality and relevancy" in Gemini Apps and Grounding with Google Search. It states explicitly that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
The practical reading: at OpenAI, opting out of training and opting out of answers are genuinely independent decisions; at Google they are not cleanly separable, because Google-Extended covers grounding as well as training. Either way, robots.txt is where the decision is made. A rule written broadly enough to keep you out of a training corpus will usually also keep you out of the answers you wanted to be in.
What genuinely influences a corpus
Nobody outside a lab can verify what entered a training set, and no publisher can measure their own contribution to one. What can be reasoned about is what a corpus is built from: large-scale crawls of public web pages, weighted toward text that is reachable, plentiful and repeated across independent sources. That yields a short list of things that plausibly help and are worth doing anyway:
- Be fetchable to the training agent — a deliberate choice, not a default. Decide it separately from the search agents.
- State facts about yourself unambiguously and identically everywhere. A model learns the version it saw most consistently, so a product described three different ways teaches three different things.
- Get the same fact stated by other sites. Independent repetition is what distinguishes a fact from a claim in a corpus, and it is not something you can produce alone.
- Publish in plain text. A fact rendered only inside an image, a video, or client-side JavaScript is a fact the corpus does not contain.
- Correct errors at the source that carries them, not only on your own site, because the erroneous version is what gets collected.
Failure modes
- Treating it as a substitute for retrievability. A memorised fact cannot be cited, corrected, or updated; a retrieved page can be all three. Retrieval is where a publisher has leverage, and training is where they have almost none. See AI search.
- Expecting a measurement. There is no published mechanism to confirm a page entered a training set. Any claim of a measured training-corpus share is unverifiable by construction.
- Blanket-blocking, then wondering why answers stopped citing you. The training decision and the answering decision share one file and get conflated constantly.
- Assuming a robots.txt change is retroactive. It governs future fetches. Content already collected is already collected, and even prospectively OpenAI documents roughly a 24-hour lag for its search crawler.
- Chasing volume. A corpus is enormous; adding pages does not move a needle you cannot see. Consistency of what the pages say is the only part of this you control.
- Mistaking a stale answer for a hallucination. A model that never saw your rebrand will describe the old brand accurately and confidently. That is a cutoff problem, and only a fetch fixes it.
Frequently asked questions
How long is the feedback loop?
At least one model generation. Published cutoffs — May 2026 for Claude Opus 5, January 2026 for Claude Sonnet 5 — bound what a trained model can know.
Which crawler is the training crawler?
GPTBot for OpenAI, Google-Extended for Gemini training — though Google-Extended also covers Gemini grounding. OpenAI's search and user-initiated agents are separate tokens.
Does opting out of training hurt search?
Google documents that Google-Extended is not a Search ranking signal and does not affect inclusion. A blanket rule that also catches the answering agents does.
Can I verify my content is in a training set?
No. There is no published mechanism for a publisher to confirm it.
Terms related to Training data optimization
The corpus a model learns from, how it differs from what an assistant retrieves at query time, and how publishers have actually responded.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
The company behind ChatGPT and the GPT model family, and — for a publisher — the operator of four separately named crawlers whose access decides whether your page can be quoted.
The AI safety company that builds the Claude models, and the three separately-named web crawlers a publisher has to answer if they want Claude to read their pages.