For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Start free

GPTBot vs OAI-SearchBot: block training, keep ChatGPT search

GPTBot crawls pages OpenAI may use to train its models. OAI-SearchBot crawls pages so ChatGPT search can show and link them. Blocking GPTBot keeps your future pages out of training and, by OpenAI's documentation, does not touch search. Blocking OAI-SearchBot does: opted-out sites are not shown in ChatGPT search answers.

Block GPTBot when you do not want your pages used to train OpenAI's models. Keep OAI-SearchBot allowed when you want ChatGPT search to show and link your pages.

Last verified
Start freeSee what AI says about you

7 days free · card required

OpenAI requests for search indexing (OAI-SearchBot)
~86%
n=7,536 OpenAI requests
OpenAI requests for training (GPTBot)
~11%
n=7,536 OpenAI requests
search crawls for every training crawl
~8:1
n=7,536 OpenAI requests
Analyst agentAnalyst · Counted these readingsattensira.com AI traffic log, read over the Attensira MCP on 2026-10-04, by the user agent each request claimed; none IP-verified · to
Answer · ChatGPTasked

Question: what is GPTBot

ChatGPT answered: A few related distinctions are useful: GPTBot ≠ ChatGPT Search crawler: GPTBot is for potential training data collection. OpenAI uses a separate crawler, OAI-SearchBot, for helping surface websites in ChatGPT search features. GPTBot ≠ a user browsing the web: When a ChatGPT user asks it to access a specific webpage, that can involve a different user agent (such as ChatGPT-User), rather than GPTBot’s automated crawling.

Sources cited

  1. 1developers.openai.com
  2. 2cdn.awstatic.com
  3. 3kitbase.dev
  4. 4muratulusoy.de
  5. 5botview.app
  6. 6growaisearch.com
  7. +4 more
Researcher agentResearcher · Read this answerStored answer, attensira.com workspace, Attensira MCP · · n=1 answer

By Karl-Gustav Kallasmaa

Updated

Researcher agentResearcher · read the AI answers shown here

Analyst agentAnalyst · counted the readings in the charts

How do GPTBot and OAI-SearchBot compare?

  • What it is

  • What the crawler is for

    GPTBot
    Training data for OpenAI's foundation models
    OAI-SearchBot
    Surfacing and linking sites in ChatGPT search

  • robots.txt user-agent token

    GPTBot
    GPTBot
    OAI-SearchBot
    OAI-SearchBot

  • Published user-agent string

    GPTBot
    ...compatible; GPTBot/1.4; +https://openai.com/gptbot
    OAI-SearchBot
    ...compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

  • Published address range list

    GPTBot
    openai.com/gptbot.json
    OAI-SearchBot
    openai.com/searchbot.json
  • What blocking it does

  • Does blocking it remove you from ChatGPT search answers?

    GPTBot
    No· No, not documented as affecting search
    OAI-SearchBot
    Yes· Yes, opted-out sites are not shown

  • Can be allowed while the other is blocked

    GPTBot
    Yes· Yes, per OpenAI
    OAI-SearchBot
    Yes· Yes, per OpenAI

  • How long a robots.txt change takes

    GPTBot
    Not stated by OpenAI
    OAI-SearchBot
    About 24 hours, per OpenAI
  • What we measured

  • Top-10k domains blocking it at the root (study of 3 Sep 2026, n=8,036)

    GPTBot
    1,147 (14.3%)
    OAI-SearchBot
    467 (5.8%)

  • Share of OpenAI requests to attensira.com, 30 days to 4 Oct 2026 (n=7,536)

    GPTBot
    About 11%
    OAI-SearchBot
    About 86%

What is GPTBot?

OpenAI's training crawler. It collects web content that may be used to train OpenAI's generative foundation models, and it is controlled by its own robots.txt token.

What is OAI-SearchBot?

OpenAI's search crawler. It indexes websites so ChatGPT's search features can surface and link them, and it is the crawler whose exclusion removes a site from ChatGPT search answers.

When should you choose GPTBot?

Block GPTBot when you do not want your pages used to train OpenAI's models. OpenAI's documentation ties GPTBot to training data, not to what ChatGPT search shows, so the block has no documented search cost.

When should you choose OAI-SearchBot?

Keep OAI-SearchBot allowed when you want ChatGPT search to show and link your pages. It is the only OpenAI crawler whose documentation states a visibility cost for blocking it: opted-out sites are not shown in ChatGPT search answers.

When is neither the right choice?

Neither token governs ChatGPT-User, the agent that fetches a page when a person asks ChatGPT to open it; OpenAI says robots.txt rules may not apply to it. And neither token touches what a model already learned before your file changed.

Where does Attensira fit?

Whatever it takes to win the answer. On Growth and Business, Attensira counts fetches by crawler at your edge next to what ChatGPT, Perplexity, Claude, Gemini and Google AI answer every day.

What sets GPTBot and OAI-SearchBot apart?

  • One company runs both, so vendor, reputation and address owner cannot tell them apart; only documented purpose and which published address list a request comes from can.
  • A wildcard group that disallows the root blocks OAI-SearchBot too unless it has its own group, which is how most accidental ChatGPT search opt-outs happen.
  • Neither token governs ChatGPT-User, the fetch a person triggers in ChatGPT; OpenAI says robots.txt may not apply to it.

Which OpenAI crawler decides whether ChatGPT search shows your site?

OAI-SearchBot. OpenAI runs several agents, and only one is documented as the reason a page can appear in ChatGPT's search features. OpenAI describes OAI-SearchBot as the crawler used to surface websites in search results in ChatGPT, and states that sites opted out of it will not be shown in ChatGPT search answers. GPTBot is documented for something else: collecting content that may be used to train OpenAI's generative foundation models.

The two are easy to confuse because one company runs both, both identify themselves with an openai.com URL, and both obey robots.txt. Confusing them produces the reverse of what most people intend. A site that wants to stay out of training and writes a blanket rule against everything from OpenAI also removes itself from ChatGPT search. A site that wants to be found and writes nothing stays in every future training crawl. OpenAI's own page says the two can be separated.

What does each crawler do with your pages?

GPTBot reads pages for a training corpus. Nothing in OpenAI's documentation connects it to what a ChatGPT answer shows today. A page GPTBot fetched may shape what a future model knows, which is a long, indirect and unmeasurable path from your site to an answer.

OAI-SearchBot reads pages so ChatGPT search can retrieve, quote and link them. That path is short and visible: when ChatGPT answers with a source list, the pages in it had to be reachable by this crawler.

A third agent, ChatGPT-User, fetches a page because a person in ChatGPT asked for it. OpenAI says robots.txt rules may not apply to it, since the request is made on a user's behalf. People often expect a GPTBot rule to cover it. It does not.

What happens if you block each one?

Blocking GPTBot keeps your future pages out of OpenAI's training crawl. It does not remove what earlier crawls collected, and OpenAI documents no effect on ChatGPT search. For a publisher whose concern is reuse of their writing, this is the lever.

Blocking OAI-SearchBot removes you from ChatGPT search answers. A company that sells to people who ask ChatGPT for recommendations should almost never do this. The exceptions are narrow: a page that should never be quoted, a staging host, or content you have contractual reasons to keep off third-party surfaces.

OpenAI says changes for OAI-SearchBot take about 24 hours to be reflected. Answers do not update on that schedule: a page ChatGPT already quoted can keep appearing, and a page you just unblocked has to be fetched again before it can be cited.

What do the robots.txt lines look like?

The common decision, out of training and into search, is two groups. Each crawler obeys the group that names it, so order does not matter.

plain text
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

If you only want part of the site kept out of training, scope the GPTBot group to paths instead of the root:

plain text
User-agent: GPTBot
Disallow: /research/
Disallow: /members/

That keeps the public marketing and documentation pages available to both crawlers while the material you care about stays out of the training crawl. It is also the version that is easiest to defend in a review, because every line names a path someone chose.

Two things go wrong often enough to check. A wildcard group (User-agent: *) that disallows the root also blocks OAI-SearchBot unless OAI-SearchBot has its own group. And robots.txt is per host: a blog on a subdomain needs its own file with the same decision in it.

How much do these crawlers actually fetch?

Our own site is the example. Attensira counts crawler requests at the edge on attensira.com, the way it does for customers on Growth and Business. In the 30 days to 4 October 2026, 7,536 requests carried OpenAI user agents, across 428 pages. About 86% were classed as search indexing (OAI-SearchBot), about 11% as training (GPTBot) and about 3% as live fetches a person asked for (ChatGPT-User).

The search crawler, not the training crawler, did most of the work. The live fetches are few and the most valuable, because each is a person reading an answer that sent them to us. None of these requests was IP-verified: the labels are the user agent each request claimed, so treat the split as a direction. And do not reconcile it with citation counts; crawling and citing are different pipelines.

How many large sites block each one?

On 3 September 2026 we read the robots.txt of the top 10,000 domains in Tranco list K9QPW and parsed 8,036. 1,147 (14.3%) disallow GPTBot at the root; 467 (5.8%) do the same to OAI-SearchBot. Most name neither: 81.4% never mention GPTBot and 90.0% never mention OAI-SearchBot, so their wildcard group decides for them.

The gap is the useful signal. Large sites have largely made the split this page recommends. A site that blocks both is now the minority case, and usually the result of a wildcard rule nobody revisited. The full table, every crawler and the method are in our study of AI crawler access.

How do you tell the two apart in your logs?

By user agent first, then by address. GPTBot sends GPTBot/1.4 with openai.com/gptbot; OAI-SearchBot sends a browser-like string ending in OAI-SearchBot/1.4 with openai.com/searchbot. Match on the token name, not the version, which changes.

The string proves nothing on its own. OpenAI publishes a separate address list for each crawler, gptbot.json and searchbot.json. A request that claims to be GPTBot from an address on neither list is not from OpenAI. If you rate-limit or block at the firewall, use those lists and refresh them on a schedule rather than copying them once, because the ranges change.

Expect the volumes to be lopsided. On our site the search crawler outnumbered the training crawler by roughly eight to one over 30 days. A log view that shows GPTBot as the busiest OpenAI agent on your site is worth a second look: either your site is unusual, or a filter is matching the wrong token.

Where do teams get this wrong?

The common mistake is the blanket rule: a team adds every AI user agent it can find to one disallow group and removes itself from ChatGPT search. The opposite is a team that wants visibility and does nothing, while a years-old wildcard group quietly disallows the root.

The third is reading a missing citation as proof of a block. Robots.txt decides whether a page can be used, never whether it will be. For the prompt "what is GPTBot", ChatGPT cited one of our pages in 12 of 33 answers and Perplexity in none of 33, with the same crawl access to both. Being readable is the floor, not the result.

Should a small site block GPTBot at all?

That is a values question more than a visibility one, and this page will not pretend otherwise. Blocking GPTBot has no documented effect on ChatGPT search, so the decision turns on whether you want your writing in a training corpus. Some companies want the opposite: a product that is described well in a model's training data may be described well in answers that never search the web. Nobody can measure that effect for a single site, and OpenAI does not claim one.

If you cannot decide, the reversible choice is to allow both today and revisit the GPTBot line when you have a reason. A robots.txt rule only governs future crawls, so waiting costs you little; what you give up by allowing GPTBot now is a few months of pages in a corpus, and you can stop that at any time. What you should not do is block OAI-SearchBot by accident while making a decision about GPTBot, because that cost shows up in answers your buyers read this week.

Does being crawled get you cited?

For the tracked prompt "what is GPTBot", ChatGPT cited an attensira.com page in 12 of 33 answers over 30 days; Perplexity cited one in none of 33. Neither named Attensira. Same crawl access, different engines, different outcomes.

ChatGPT answers citing attensira.com (12)
36.4%
n=33 answers
Perplexity answers citing attensira.com
0%
n=33 answers
answers naming Attensira, both engines
0%
n=66 answers
Analyst agentAnalyst · Counted these readingsAnswers to one tracked prompt, read over the Attensira MCP on 2026-10-04, attensira.com workspace · to

What about ChatGPT-User and the other OpenAI agents?

OpenAI documents more than these two agents, and the one that matters most after OAI-SearchBot is ChatGPT-User. It fetches a page at the moment a person in ChatGPT asks to open it, or when a custom GPT does so on their behalf. On attensira.com it was about 3% of OpenAI's requests in 30 days, the smallest share and the one with a reader attached to every request.

Because the request is made for a user, OpenAI says robots.txt rules may not apply to it. That has two consequences for this decision. A GPTBot block never reaches it, so a site that blocked GPTBot can still see ChatGPT-User in its logs and should not read that as the block failing. And if you genuinely need to stop live fetches, for a members area or a paywall, the tool is authentication or a firewall rule on the published addresses, not a line in robots.txt.

How do you check that a change worked?

Fetch your live robots.txt from the public host, not your repository. Wait the day OpenAI gives, then look for OAI-SearchBot in your logs, checked against the published list. If you blocked GPTBot, its requests should fall to robots.txt fetches only.

Then look at answers, not just logs. Ask ChatGPT what your buyers ask and see whether your pages appear in the sources. One reading is noise; a daily reading of the same questions over a few weeks is a measurement.

Questions people ask

Does blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search, by OpenAI's own documentation. GPTBot collects content that may be used to train OpenAI's foundation models, and OpenAI states that a site can allow OAI-SearchBot while disallowing GPTBot. What decides whether ChatGPT search can show and link you is OAI-SearchBot: sites opted out of it are not shown in ChatGPT search answers.
Is content fetched by OAI-SearchBot used to train OpenAI's models?
OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT's search features and lists it separately from GPTBot, the training crawler. The documented way to keep pages out of training while staying in search is to allow OAI-SearchBot and disallow GPTBot, each in its own user-agent group.
How long does a robots.txt change take to apply?
OpenAI says changes for OAI-SearchBot take about 24 hours to be reflected. Plan for a day before you check, and confirm in your server logs rather than in an answer: answers can keep quoting a page from before the change.
Does a GPTBot rule also stop ChatGPT-User?
No. ChatGPT-User is the agent that fetches a page when a person in ChatGPT, or a custom GPT, asks for it, and it has its own token. OpenAI says robots.txt rules may not apply to it, because the request is made on a user's behalf. Stopping it takes a server or firewall rule, not a GPTBot line.
How do I know a request really came from OpenAI?
Check the source address against the list OpenAI publishes for that crawler: openai.com/gptbot.json for GPTBot and openai.com/searchbot.json for OAI-SearchBot. A user-agent string alone proves nothing, because any client can send one.
How often do these crawlers visit a small site?
On attensira.com, 7,536 requests carried OpenAI user agents in the 30 days to 4 October 2026, across 428 pages. About 86% were classed as search indexing, 11% as training and 3% as live fetches a person asked for. The agents are self-reported and none was IP-verified, so read the split as a direction, not an audit.

Where do these facts come from?

16 sourced claims, each with the page it was read on and the day.

  1. OpenAI documents GPTBot as the crawler used to collect content that may be used to train its generative foundation models.

    developers.openai.com · retrieved

    “Used to crawl content that may be used in training our generative AI foundation models”

  2. GPTBot is controlled with the robots.txt user-agent token GPTBot.

    developers.openai.com · retrieved

  3. OpenAI publishes GPTBot's user-agent string as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

    developers.openai.com · retrieved · changes often, check the source

  4. OpenAI publishes GPTBot's address ranges as a machine-readable list at openai.com/gptbot.json

    developers.openai.com · retrieved

  5. OpenAI's documentation ties GPTBot to inclusion in training datasets rather than to what appears in ChatGPT search results.

    developers.openai.com · retrieved

  6. OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT's search features.

    developers.openai.com · retrieved

    “Used to surface websites in search results in ChatGPT's search features”
Show all 16 sources

  1. OAI-SearchBot is controlled with the robots.txt user-agent token OAI-SearchBot.

    developers.openai.com · retrieved

  2. OpenAI publishes OAI-SearchBot's user-agent string as Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

    developers.openai.com · retrieved · changes often, check the source

  3. OpenAI publishes OAI-SearchBot's address ranges as a machine-readable list at openai.com/searchbot.json

    developers.openai.com · retrieved

  4. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

    developers.openai.com · retrieved

    “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”

  5. OpenAI states that robots.txt changes for OAI-SearchBot take about 24 hours to be reflected.

    developers.openai.com · retrieved

  6. OpenAI states that a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot.

    developers.openai.com · retrieved

  7. Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 1,147 (14.3%) disallow GPTBot at the site root.

    attensira.com · retrieved · changes often, check the source

  8. Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 467 (5.8%) disallow OAI-SearchBot at the site root.

    attensira.com · retrieved · changes often, check the source

  9. In the attensira.com workspace, read through the Attensira MCP on 4 October 2026, about 11% of 7,536 requests with OpenAI user agents in the 30 days to 4 October 2026 were classed as training (GPTBot), by the user agent each request claimed; none was IP-verified.

    attensira.com workspace, read over the Attensira MCP on · changes often, check the source

  10. In the attensira.com workspace, read through the Attensira MCP on 4 October 2026, about 86% of the same 7,536 OpenAI requests in the 30 days to 4 October 2026 were classed as search indexing (OAI-SearchBot), across 428 distinct pages.

    attensira.com workspace, read over the Attensira MCP on · changes often, check the source

Put the crew to work on your brand, 7 days free.

Start freeSee what AI says about you

7 days free · card required