GPTBot vs OAI-SearchBot: which OpenAI bot decides ChatGPT visibility

The short answer

GPTBot and OAI-SearchBot are both operated by OpenAI, and they do opposite jobs. GPTBot collects content that may train OpenAI's foundation models. OAI-SearchBot indexes content so ChatGPT's search features can surface and link it. Only the second one has a documented cost when you block it.

That distinction is the whole page. Almost every robots.txt mistake in this area comes from treating "the OpenAI bot" as one thing and writing a single rule that either gives away more than intended or removes the site from ChatGPT search answers by accident.

Why one company runs two crawlers

Training and retrieval are different products with different legal and commercial shapes, and OpenAI has separated them at the level of the user agent. GPTBot's documented job is collecting content that may be used in training generative foundation models. OAI-SearchBot's documented job is surfacing websites in ChatGPT's search features — retrieval at answer time, with a link back to the source.

The separation is not cosmetic. Each crawler has its own robots.txt token, its own user-agent string, and its own published list of address ranges. OpenAI's crawler documentation lists four agents in total, GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, each with its own token.[^openai-bot-count] A directive written for one of them has no effect on the other three. That is unusually clean compared with operators who fold every purpose into a single agent, and it means a publisher can express a nuanced position — no training, yes retrieval — that a single-token operator makes impossible.

What each one costs you if you block it

The honest asymmetry: blocking GPTBot has no documented effect on whether ChatGPT search can show your pages, and blocking OAI-SearchBot does.

OpenAI's documentation ties GPTBot to inclusion in training datasets. If your objection is to a commercial model being trained on your archive, GPTBot is the token that expresses it, and you can disallow it without a documented penalty to how ChatGPT's search surfaces treat you. Whether the model already knows your content from earlier crawls is a separate question that no robots.txt directive can retroactively answer, and nobody should promise you otherwise.

OAI-SearchBot is the opposite case. OpenAI states plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. That is not an inference from behaviour, it is the operator's own published statement, and it is the single most consequential sentence in OpenAI's crawler documentation for anyone who cares about being cited. If ChatGPT is a channel you want, this token stays allowed.

There is a third position worth stating because it is frequently the correct one: block neither. A product site, a documentation site or a marketing site exists to be found. The training objection that motivates blocking GPTBot is a real and defensible position for original journalism, paywalled research and creative archives whose value is the work itself. It is a much weaker position for a company whose problem is that not enough people know it exists.

The robots.txt directives, copyable

Disallow training, keep ChatGPT search visibility. This is the configuration most publishers actually want:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Refuse both, accepting that ChatGPT search will not show your pages:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

Allow both, which is the default state of any site with no rules for these tokens. Writing it out is still worth doing, because it documents a decision rather than an oversight:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Two mechanical notes. Robots.txt matching is on the token, not the full user-agent string, so a user-agent line naming GPTBot is correct and one pasting the whole Mozilla string is not. And rules apply per host and per scheme, so a directive on your apex domain does nothing for a documentation subdomain that serves its own robots file.

Telling them apart in your logs

Both crawlers announce themselves. GPTBot sends a user agent ending in compatible; GPTBot/1.4; +https://openai.com/gptbot, and OAI-SearchBot sends one ending in compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. The self-identifying URL is the reliable part of the string to filter on, because it does not change when the version number does.

That version number is a real trap. Both agents currently publish a /1.4 version, and a log filter or firewall rule written against GPTBot/1.0 will silently stop matching the moment OpenAI increments it. Match on the token and the identifying URL, never on the whole string.

User-agent strings can be forged by anyone, so the second half of verification is the address. OpenAI publishes per-crawler ranges at openai.com/gptbot.json and openai.com/searchbot.json. A request claiming to be OAI-SearchBot from an address outside that list is somebody else's scraper wearing OpenAI's name, and you should treat it accordingly rather than counting it as evidence that ChatGPT is indexing you.

Where publishers get this wrong

The most common failure is a blanket block. Somebody reads a headline about models training on the open web, disallows GPTBot, then adds OAI-SearchBot to the same block to be safe — and quietly removes the site from ChatGPT search answers in the same commit. The intention was to refuse training. The effect was to refuse distribution.

The second failure is the reverse: assuming that because ChatGPT can talk about your company, your robots file does not matter. Model knowledge from training and retrieval at answer time are separate paths, and the second one is the one you can influence this quarter.

The third is treating robots.txt as enforcement. It is a published request that well-behaved operators honour. OpenAI documents its crawlers as robots-controlled, and the published address lists exist precisely so that you can check whether the request is being honoured. If you need enforcement rather than a request, that is a firewall rule, and it is a different piece of work with different costs.

Subdomains, staging hosts and the rules you forgot

Robots.txt is per host. That sentence causes more accidental exposure and more accidental invisibility than any other property of the standard, and it bites hardest at exactly the companies who care most about this pairing.

A typical developer-tools company serves marketing copy from its apex domain, product documentation from a documentation subdomain, a changelog from a third host and a staging environment from a fourth. Each of those hosts serves its own robots file, and each therefore carries its own answer to the GPTBot and OAI-SearchBot question. The documentation host is usually the one that matters most for citation — it is where the specific, extractable, answer-shaped content lives — and it is also the host most likely to have been set up by a framework default that nobody has revisited.

The failure mode is asymmetric in an unhelpful direction. A blanket disallow copied onto a staging host is harmless. A blanket disallow copied onto the documentation host removes the most citable pages you own from ChatGPT search while the marketing site, which nobody wants to quote, stays perfectly visible. Audit every host you serve, not just the one in your browser bar.

The same applies to path scoping. Disallowing a path prefix for OAI-SearchBot is a narrower and often smarter move than disallowing the host: pricing pages, customer lists and gated resources can be excluded while documentation stays open. Robots.txt path rules apply to whichever token they sit under, so a path disallow written under GPTBot does nothing to OAI-SearchBot and vice versa.

What neither token controls

It is worth being precise about the limits, because overpromising here is how people end up disappointed. Neither of these directives controls what a model already learned. Neither controls whether some third party has reproduced your content on a site with its own robots file. Neither controls what a user pastes into a chat window. And neither is a copyright position — a robots directive is a machine-readable preference, not a licence.

There is also a question these tokens genuinely cannot answer: whether being crawled turns into being cited. Access is necessary and not sufficient. OAI-SearchBot being allowed to fetch a page is the precondition for that page appearing in a ChatGPT search answer; it is not a promise that it will. The work that follows — making the page answer a real question, in extractable form, better than the pages currently being quoted — is a content problem, not a robots problem.

Verifying that your change did anything

A robots.txt edit is a statement of intent. What follows it in your server logs is the fact. After changing either directive, the question worth answering is whether the agent you disallowed stopped fetching and the agent you allowed kept fetching — per URL, not in aggregate, because a rule scoped to a path behaves differently from one scoped to the host.

Attensira's crawler logs exist for exactly this: they record which AI agent fetched which URL and when, so a change to GPTBot or OAI-SearchBot access can be confirmed against observed behaviour rather than assumed. If you only need to check the current state of a file, the robots.txt generator and the bot access score will tell you what your rules currently permit.

Related comparisons

The same training-versus-retrieval split appears at every major operator, with different defaults. Anthropic separates its training crawler from its user-initiated fetcher in ClaudeBot vs Claude-User. Perplexity's split is sharper still, because one of its two agents is documented as generally ignoring robots.txt — see PerplexityBot vs Perplexity-User. For OAI-SearchBot against its closest competitor rather than its sibling, read OAI-SearchBot vs PerplexityBot. And for how GPTBot compares with the open-dataset crawler that predates it, see CCBot vs GPTBot.