Claude-SearchBot vs OAI-SearchBot: the two crawlers behind assistant answers

The short answer

If you only ever check two robots.txt tokens, check these. Claude-SearchBot and OAI-SearchBot are the crawlers that stand between your pages and the answers two of the most-used assistants produce, and both operators say in their own documentation that blocking them has a cost.

The cost is worded differently, and the difference is instructive. Anthropic describes blocking Claude-SearchBot as reducing site visibility and accuracy in user search results — a degradation. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links — an exclusion with a small consolation attached. One vendor tells you the picture gets blurrier. The other tells you the picture no longer contains you.

Why these are not the training crawlers

Both vendors run at least three agents, and the separation is deliberate. Anthropic documents ClaudeBot for training, Claude-User for fetches a person asked for, and Claude-SearchBot for search quality. OpenAI documents four agents, each with its own robots.txt token: GPTBot for training, ChatGPT-User for user actions, OAI-AdsBot for ad-safety checks, and OAI-SearchBot for search.

That structure exists so a publisher can hold a nuanced position. No training, yes retrieval, is a coherent stance and both vendors have built the tokens to express it. What they cannot protect you from is a robots.txt written as if "the OpenAI bot" and "the Anthropic bot" were single things. A blanket disallow aimed at the training question sweeps up the retrieval crawler in the same commit, and the site owner finds out months later, from a competitor's citation, that they opted out of the only channel they wanted.

The tell is simple. Read your robots file and ask, for every AI-vendor token in it, which question that line was answering. If a line was written about training and it names a SearchBot token, it is in the wrong place.

The asymmetry in what each operator promises

Anthropic's phrasing is honest about a thing most operators leave implicit: blocking a search crawler does not necessarily make you invisible, it makes the assistant's picture of you worse. Models carry knowledge from other sources. What the search crawler adds is currency and accuracy — the pricing that changed last month, the API that was deprecated, the product you renamed. Refuse it and the assistant keeps talking about you, using whatever it has. That is frequently a worse outcome than silence, because inaccurate representation is harder to notice than absence and harder still to correct.

OpenAI's phrasing is sharper and, for anyone measuring citations, more consequential. An opted-out site is not shown in ChatGPT search answers. The clause about navigational links is not a loophole worth planning around: a bare link is not a citation, it does not carry your framing, and it does not put a sentence of your documentation in front of somebody deciding what to buy.

Neither statement is a promise in the other direction, and it is worth saying plainly. Allowing these crawlers is necessary for citation and nowhere near sufficient. Access gets you into the pool of candidate sources. Whether a given page gets quoted depends on whether it answers a real question in extractable form better than the pages currently being quoted, which is a content problem no robots directive touches.

Verification, and the difference in how the two publish

Both operators publish crawler addresses, which already puts them ahead of most of the web's crawler traffic, and they publish them differently.

OpenAI maintains a per-crawler file, and openai.com/searchbot.json covers OAI-SearchBot alone. That means an address check is by itself sufficient to prove a particular request came from the search crawler rather than the training crawler. If you need to demonstrate to somebody else — an SEO lead, a security reviewer, a board deck — that ChatGPT's search infrastructure is reading your documentation, that file is the evidence.

Anthropic publishes at claude.com/crawling/bots.json, one file covering its crawlers. The address check proves the request came from Anthropic; the user-agent token is what tells you which agent it was. Since user agents are forgeable by anyone, an Anthropic fetch is verified in two steps of unequal strength. In practice this is fine — nobody is forging a Claude-SearchBot header from inside Anthropic's own address ranges — but it does mean a log query that separates Anthropic's training traffic from its search traffic is trusting a string.

A version note for the OpenAI side: the published OAI-SearchBot user agent carries a /1.4 version number today. Filters written against a specific version quietly stop matching after a bump, and a filter that stops matching looks exactly like a crawler that stopped visiting. Match on the token and the self-identifying URL.

The robots.txt lines

Allow both retrieval crawlers, refuse both training crawlers. This is the configuration a company that wants to be cited but objects to training should be running:

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Throttle Anthropic's search crawler rather than blocking it, using the directive Anthropic documents as supported:

User-agent: Claude-SearchBot
Crawl-delay: 5

Two mechanical points. Robots.txt matches on the token, so User-agent: OAI-SearchBot is right and pasting the full Mozilla string is not. And rules are per host: the documentation subdomain, the changelog host and the help centre each serve their own robots file and therefore each carry their own answer. For a technical product the documentation host is usually where the citable content lives and usually the host whose robots file was generated by a framework default nobody revisited.

Where the damage usually happens

The most common failure is a well-meant blanket block. Somebody reads a headline about training, disallows every AI-vendor token they can find, and removes the company's documentation from two assistants' answer surfaces in a single commit. Intention: refuse training. Effect: refuse distribution.

The second is the mirror image — assuming that because an assistant can already talk about your company, the robots file is irrelevant. Model knowledge and retrieval at answer time are different paths. The second one is the one you can change this quarter, and it is the one that carries a link.

The third is scope. A rule on the apex domain while docs. serves an untouched framework default is the single most common way a company is both convinced it has opted in and effectively opted out. Audit every host you serve.

Which one to worry about first

If you have limited attention — and everyone does — the honest ranking is by where your buyers already are, not by which vendor you find more interesting.

For a developer-tools company, Claude-SearchBot deserves the first look. Claude is increasingly used inside editors and terminals, where the assistant is the reading surface and the developer never opens a browser tab at all. In that setting a documentation page that cannot be fetched is a page that cannot be quoted, and there is no organic search result waiting downstream to catch the miss. The pages that matter are reference docs, error-message pages and quickstarts, and they are almost always on a subdomain with its own robots file.

For a company selling to a general business audience, OAI-SearchBot is the sharper risk, for the simple reason that OpenAI documents exclusion rather than degradation. The downside is discrete and stated: you are not shown in the answers. There is no partial credit to fall back on beyond a navigational link.

For most companies the correct answer is that both are allowed, the question takes ten minutes to settle, and the ten minutes are worth spending because the failure mode is invisible. Nothing breaks when a retrieval crawler is blocked. Traffic does not drop in a way that shows up as an incident. You simply stop being one of the sources an assistant reaches for, and you find out from somebody else's citation.

What neither token controls

Neither crawler directive governs what a person pastes into a chat window; both vendors document separate user-triggered agents for that traffic, with different rules. Neither reaches copies of your content that a third party hosts under their own robots file. Neither is a licence or a legal position — a robots directive is a machine-readable preference to a well-behaved client, nothing more. And neither retroactively removes anything a model already learned.

It is also worth reviewing the decision on a cadence. Both of these pages change without announcement: OpenAI's crawler roster has grown and each addition arrives with a new token, and Anthropic's article now describes three agents where it once described the crawler as a single thing. A robots file written to be exhaustive last year is merely partial today, and the failure is silent — a new agent under a name your file has never heard of simply crawls, doing a job you thought you had already decided about.

Confirming access, per host and per URL

A robots.txt edit states an intention. The access log records what happened. After changing either token, the question is whether the crawler you allowed is actually fetching the pages you care about — not the homepage, the pages that answer questions.

Attensira's crawler logs record which AI agent fetched which URL and when, which is how "is Claude reading our docs" stops being a guess. To see what your current file permits before you touch it, the robots.txt generator and the bot access score will read it back to you.

Related comparisons

For each vendor's internal split rather than the rivalry between them, read GPTBot vs OAI-SearchBot and ClaudeBot vs Claude-User. For the two training crawlers head to head, see ClaudeBot vs GPTBot. And for OAI-SearchBot against the search crawler of the assistant built entirely around citation, see OAI-SearchBot vs PerplexityBot.