Bots
What is GPTBot, and should you allow it?
GPTBot is OpenAI's training crawler: OpenAI documents it as collecting content that may be used to train its foundation models. Blocking it does not remove you from ChatGPT, because ChatGPT's search results come from a separate agent, OAI-SearchBot. Block GPTBot if you object to training use; allowing it earns no citations either way.
7 days free · card required
830GPTBot requests to attensira.com
Estimated: 11% of OpenAI's 7,536 requests · #3 of 4 training agents
| Agent | Share of requests | Requests |
|---|---|---|
| Meta-ExternalAgent | 131,105 | |
| Bytespider | 3,505 | |
| GPTBot (this page) | ≈830 | |
| ClaudeBot | ≈68 |
≈: an operator's requests split by the purpose each of its agents declares. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP · · n=143,601 requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block GPTBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves GPTBot today.
# Block OpenAI's training crawler, keep ChatGPT's search# crawler.User-agent: GPTBotDisallow: / User-agent: OAI-SearchBotAllow: /User-agent: GPTBotAllow: /Disallow: /api/We name GPTBot and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs GPTBot, and what is it for?
OpenAI runs it, and we file it under training. Collects text that may enter a training corpus. Blocking it removes no page from any assistant's search index or citations, though a model can still name you from what it already learned.
From OpenAI’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Does blocking GPTBot remove you from ChatGPT?
No. Confusing the two is how a site meaning to opt out of training ends up opting out of ChatGPT search. OpenAI runs four agents, each with its own robots.txt token and its own documented job. GPTBot's job is to crawl content that may be used in training OpenAI's foundation models1. Surfacing websites in ChatGPT's search results is the job of a different agent, OAI-SearchBot5. Fetching one page because a person just asked about it is the job of a third, ChatGPT-User, and OpenAI says robots.txt rules may not apply to that one because the fetch follows a user action6.
So a rule that names GPTBot does exactly one thing: it keeps your text out of the training crawl. Your pages stay findable in ChatGPT search. A training fetch never ends in a link, a referral or anything your analytics can see, so blocking GPTBot costs you no citations or referrals, and allowing it earns you none.
The way sites lose ChatGPT by accident is the wildcard. A Disallow: / under User-agent: * applies to every agent without a group of its own. If OAI-SearchBot has no group of its own in that file, the wildcard block reaches it too.
Does a GPTBot group override your wildcard group?
Yes, completely. RFC 9309 says a crawler obeys the User-agent: * group only when no group matches its own token7. A file with User-agent: * and Disallow: /, plus a separate User-agent: GPTBot group with Allow: /, lets GPTBot crawl everything. GPTBot reads its own group and ignores the wildcard.
That cuts both ways. Once GPTBot has a group, none of the wildcard's rules reach it, so a private path you disallowed under * is open to GPTBot unless you repeat the rule in its group. The combining rule in the standard is narrower than it sounds: two groups that both name GPTBot are read as one. A GPTBot group and a wildcard group never are.
So name every agent you have an opinion about, as the block in the robots.txt section above does for GPTBot and OAI-SearchBot, and put the full set of rules you want for each one in its own group.
How do I verify that a visit really came from GPTBot?
A user-agent header is free text, so a log line naming GPTBot is a claim until you check it.
Match the visiting IP against the published ranges in the record.
OpenAI publishes a separate range file per agent, so a verified request identifies which of the four agents made it, independently of the header it sent.
What does GPTBot look like in a real site's logs?
Here is our own site. Attensira measures AI crawler requests at the edge on attensira.com, the same way the product does for a customer. In the 30 days to 4 October 2026 it recorded 7,536 requests from OpenAI's agents across 428 pages. Three of the four agents appeared: GPTBot, OAI-SearchBot and ChatGPT-User.
By the purpose each agent declares, 86% were search indexing, 11% were training and 3% were fetches for a live answer (shares of n=7,536). GPTBot, the training crawler, was the 11%, about 830 requests. Most of what OpenAI fetched from us was for search, which is the traffic a GPTBot-only rule never touches.
Two cautions. These counts are user agents as sent; the measurement does not check each request against OpenAI's range file, so read them as direction, not proof. And one site's mix is one site's mix: a large publisher's ratio will look different.
How is GPTBot different from OAI-SearchBot and ChatGPT-User?
GPTBot
- Documented job
- Crawls content that may be used for training
- What blocking it does
- Keeps your text out of that crawl. No effect on citations
OAI-SearchBot
- Documented job
- Surfaces websites in ChatGPT's search results
- What blocking it does
- Removes you from the pool ChatGPT search cites from
ChatGPT-User
- Documented job
- Fetches a page for a user's action in ChatGPT
- What blocking it does
- Robots.txt may not apply to it, per OpenAI
The side-by-side, with the robots.txt for each combination, is in GPTBot vs OAI-SearchBot. If you are also weighing the open crawl corpus that many models train on, CCBot vs GPTBot covers that pair.
- IP range file
- gptbot.json
OpenAI publishes no reverse-DNS suffix for GPTBot; verify by the range file.
ResearcherTranscribed from OpenAI documentation · Agents filed beside GPTBot
How often do sites block GPTBot?
Measured by us, not reported by OpenAI: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow GPTBot at the root
- 14.3%
- 1,147 of 8,036 domains
- allow it by name
- 4.3%
- 349 domains
- never mention it
- 81.4%
- 6,540 domains
Why match the token and not the version?
OpenAI documents GPTBot's full header with a version number, GPTBot/1.4 as of the date in the user-agent section above2. The number increments. A log filter or firewall rule pinned to GPTBot/1.4 stops matching the next time OpenAI ships, and it fails silently: the requests keep arriving and your report simply shows fewer of them. Match the bare token GPTBot, case-insensitively, as a whole word.
What can Attensira show you about GPTBot?
Two separate things, kept apart on purpose. AI traffic, on the Growth and Business plans, counts which pages AI crawlers fetch, grouped by operator and by declared purpose, measured in edge middleware because crawlers do not run JavaScript. Prompt tracking reads what ChatGPT and the other engines actually answer about your brand every day. A page can be crawled constantly and never cited, so a GPTBot spike tells you nothing about ChatGPT answers. The answers themselves do.
If you want the numbers for your own site, crawler logs shows the fetches and the robots.txt generator writes the rule.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https:/Should I allow GPTBot in my situation?
Concrete cases, and what each one implies for this agent.
You object to your writing being used to train commercial models, but you want ChatGPT to be able to find and link your pages.
Disallow GPTBot by name. Leave OAI-SearchBot allowed.
These are two separate robots.txt tokens with two separate documented jobs. GPTBot's is training; OAI-SearchBot's is surfacing websites in ChatGPT's search results. A rule naming only GPTBot satisfies the training objection and leaves the citation path open.
You wrote a wildcard rule to keep AI companies out and your ChatGPT referrals disappeared.
Give OAI-SearchBot a group of its own that allows it, which takes it out of the wildcard entirely, and keep the block you meant in a named GPTBot group.
A rule under the wildcard user agent applies only to agents without a group of their own. Unless the file names them, that includes the search crawlers and the user-triggered fetchers as well as the training crawler you were aiming at. RFC 9309 says a crawler obeys the * group only when no group matches its token, so one named group is the whole exemption.
Your logs show a spike of GPTBot traffic hammering a section of the site, and you want to know whether it is real before you rate-limit it.
Check the source addresses against openai.com/gptbot.json before doing anything else.
The user-agent header is free text, and AI crawler tokens are worth copying because sites make exceptions for them. If the addresses are not in OpenAI's published range, you are rate-limiting a scraper wearing OpenAI's name, and the robots.txt rule you were about to write would have had no effect on it at all.
What else do people ask?
Does GPTBot respect robots.txt?
How do I know a GPTBot request in my logs is really GPTBot?
How much of OpenAI's traffic is GPTBot?
Where do these facts come from?
7 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
OpenAI documents GPTBot as the agent used to crawl content that may be used in training its generative AI foundation models.
developers.openai.com · retrieved
“crawl content that may be used in training our generative AI foundation models”
OpenAI documents GPTBot's full user agent as "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot".
developers.openai.com · retrieved · changes often, check the source
OpenAI states that GPTBot respects robots.txt.
developers.openai.com · retrieved
OpenAI publishes the IP ranges GPTBot fetches from as a machine-readable file at openai.com/gptbot.json, one of four per-agent range files.
developers.openai.com · retrieved
OpenAI documents OAI-SearchBot, a separate robots.txt token from GPTBot, as the agent used to surface websites in search results in ChatGPT's search features.
developers.openai.com · retrieved
“surface websites in search results in ChatGPT's search features”
OpenAI documents ChatGPT-User as the agent used for certain user actions in ChatGPT and Custom GPTs, and states that robots.txt rules may not apply to it because the fetch follows a user action.
developers.openai.com · retrieved
RFC 9309 section 2.2.1 states that a crawler obeys the group with a * user agent only when no group matches its own product token. Its rule for combining groups covers several groups that match the same token.
rfc-editor.org · retrieved
“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”
Want whatever it takes to win the answer?
7 days free · card required