Amzn-SearchBot vs OAI-SearchBot: two search crawlers, two default behaviours

The short answer

Amzn-SearchBot and OAI-SearchBot do the same category of work — crawl the open web so a product can answer questions using it — and both operators state that this traffic does not feed model training. On the substance they are close to interchangeable.

The interesting difference is what happens when you say nothing. Amazon documents that if your robots.txt omits Amzn-SearchBot but permits other search bots, it will crawl in accordance with the directives you gave those other search bots. OpenAI documents no such inheritance for OAI-SearchBot: the token governs itself, and silence means the default state, which is allowed.

One of these crawlers reads rules you wrote for somebody else. That is the finding, and it has consequences in both directions.

The inheritance rule, and why it cuts both ways

Amazon's fallback is, on its face, a courtesy. Most robots.txt files were written years before Amazon's search crawler existed, and rather than treating that silence as permission to do whatever it likes, Amazon looks at how you treat comparable agents and behaves accordingly. A site that welcomes search crawlers gets crawled. A site that is restrictive with them gets the same restriction.

For a permissive site this is fine and slightly helpful — you never had to think about it, and the outcome matches your intent.

For a restrictive site it is also fine, and this is the half worth sitting with. If your robots.txt carries a narrow Disallow under a major search engine's token — because of a crawl-budget experiment in 2019, a staging incident, or a rule somebody added to protect a heavy faceted-search path — Amazon's crawler is following that rule today. Nobody decided that. It is an inherited default, and the only place it is visible is in your access logs, where the absence of a crawler is the hardest thing in the world to notice.

The fix is not complicated. If you care about Amazon's search surfaces, name the token explicitly rather than letting it inherit. An explicit rule is also self-documenting for the next person who reads the file.

OAI-SearchBot has no such behaviour documented. Its token governs its own crawling, and a file that says nothing about it leaves it in the default allowed state. Simpler, and simpler in the direction that favours visibility.

What each operator says the crawl is for

Amazon's entry is specific about the destination: content crawled by Amzn-SearchBot is eligible to appear in search experiences such as Alexa. That is a different audience from the one most GEO discussions assume. Voice queries and Amazon product surfaces skew towards shopping, local questions, quick facts and how-to answers, and away from the technical evaluation queries that dominate an assistant used inside an editor.

OpenAI's entry names the destination and the cost of refusing it in the same breath: OAI-SearchBot surfaces websites in ChatGPT's search features, and sites opted out of it will not be shown in ChatGPT search answers, though can still appear as navigational links. That parenthetical is not a consolation worth planning around. A bare navigational link carries no framing, no quoted sentence and no context — it is not a citation.

Both operators keep training on a different token. Amazon writes the negative into the Amzn-SearchBot entry directly, stating that it does not crawl content for generative AI model training and documenting Amazonbot separately for content that may train Amazon AI models. OpenAI achieves the same separation structurally: GPTBot is its own token with its own documentation, and disallowing it indicates a site's content should not be used in training generative AI foundation models.

Verification is genuinely good on both sides

Both operators publish a separate address file per crawler, which is the strongest verification posture available.

That per-crawler granularity matters more than it sounds. When an operator publishes one address list covering everything it runs, an address check tells you the request came from that company but not which agent made it, and you fall back on a user-agent string that anyone can forge. With per-crawler files, an address check alone distinguishes the search crawler from the training crawler. If you need to demonstrate to a security reviewer or a sceptical colleague that ChatGPT's search infrastructure — specifically, not OpenAI in general — is reading your documentation, the evidence exists.

The version numbers are the usual trap and they are unusually asymmetric here. Amazon publishes Amzn-SearchBot/0.1; OpenAI publishes OAI-SearchBot/1.4. Neither number is stable, and any filter, WAF signature or dashboard built on the full string stops matching the moment either increments — silently, presenting as a crawler that lost interest in your site. Match on the bare token.

The robots.txt lines

Name both explicitly, which is the configuration that removes the inheritance question entirely:

User-agent: Amzn-SearchBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Refuse training at both vendors while keeping both search surfaces, since each operator keeps training on a different token:

User-agent: Amazonbot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

And the shape to look for in an existing file, because this is what inheritance looks like in the wild — a narrow rule written for one search engine, silently governing Amazon too:

User-agent: Googlebot
Disallow: /search/

Two mechanics. Robots.txt matches on the token, so the User-agent: line takes Amzn-SearchBot and not the whole Mozilla string. And rules are per host and per scheme: the apex domain, the documentation subdomain and the help centre each serve their own file, and for most technical companies the documentation host holds the most citable content and receives the least review.

Which one to prioritise

If you sell to businesses, to developers or to anyone who evaluates a product by reading, OAI-SearchBot is the one to check first. The consequence is stated, the surface is large, and the failure mode is discrete: you are not in the answers.

If you sell physical goods, serve a local market, or publish the kind of quick factual content voice assistants reach for, Amazon's surfaces deserve real attention and almost never get it. Amzn-SearchBot appears in a fraction of the robots files that name OpenAI's tokens, and the inheritance rule means many of those silent files are already handing Amazon a rule written for somebody else.

For most companies both are allowed, the review takes ten minutes, and the ten minutes are worth it because both failure modes are invisible. Nothing breaks when a search crawler is excluded. You simply stop being a candidate source, and you find out from a competitor's citation.

Auditing a file for inherited rules

The inheritance behaviour makes one review job worth doing properly, and it takes about twenty minutes for most sites.

Open every robots.txt you serve — apex, www if it differs, docs, help, blog, changelog, status, app — and for each one write down two lists: the tokens named explicitly, and the directives attached to search-engine tokens. The second list is what Amazon's crawler may be following on your behalf.

What people find is rarely a deliberate policy. It is a Disallow: /search/ added to stop a faceted-navigation crawl trap, a Disallow: /api/ from a security review, a Disallow: / on a staging host that later became a production subdomain, or a block written during an incident and never lifted. Each of those was a decision about one crawler at one moment, and each is now a decision about Amazon's search surfaces too.

The remedy is to name the tokens you care about explicitly, which converts an inherited default into a stated position. That is worth doing even where the inherited outcome happens to be the one you want, because a rule that expresses itself survives a rewrite and an inherited one does not.

A note on why these decisions get made badly

Crawler policy tends to be set in the worst possible conditions: during an incident, by whoever is on call, under time pressure, with a blunt instrument. Somebody sees traffic they do not recognise, adds a disallow, resolves the incident and moves on. The rule outlives the incident by years.

The structural fix is to treat robots.txt like any other production configuration. Put it in the repository next to the code, require a reviewer, and demand a comment on every rule saying what question it answers and when it was last checked against the operator's documentation. A file with that discipline is a file where inheritance cannot happen by accident, because every token that matters is named and every rule explains itself.

It also makes the quarterly review tractable. Both of these operators have added agents over time, and each addition arrives with a token your file has never heard of. The review is not "is our policy still right" — it is "has anyone shipped a crawler we have not decided about", which is a five-minute diff when the file is documented and an afternoon of archaeology when it is not.

What allowing them does not buy

Access is the precondition for citation and not a substitute for it. Both crawlers will fetch pages neither product ever quotes, because the selection happens downstream against everything else available on the question. The work that closes that gap is content work — answering a question somebody actually asks, in a form that lifts cleanly into an answer — and no robots directive shortens it.

The limits are the usual ones and worth restating. Neither directive is retroactive. Neither reaches copies of your content that a third party hosts under their own robots file. Neither governs what a person pastes into a chat window; both operators document separate user-triggered agents for that traffic, with different rules. And neither is a licence — a robots directive is a machine-readable preference to a well-behaved client, not a contract.

Confirming access, per host and per URL

A robots.txt edit is an intention. Your access log is the outcome, and with both operators publishing per-crawler address files, these two agents can be verified end to end: match the token, check the address, and confirm which URLs were fetched.

The last part is where the value is. Knowing a search crawler reached your homepage tells you almost nothing. Knowing whether it has read the pages that answer your buyers' questions tells you whether you are a candidate source at all. Attensira's crawler logs record which AI agent fetched which URL and when. To read your current rules — including which agents your file does not mention, which is where inheritance hides — the robots.txt generator and the bot access score will tell you what your file permits today.

Related comparisons

For OpenAI's own split between the training crawler and the search crawler, read GPTBot vs OAI-SearchBot. For OAI-SearchBot against the crawler behind an answer engine built entirely on citations, see OAI-SearchBot vs PerplexityBot. For the two largest assistant vendors' search crawlers, see Claude-SearchBot vs OAI-SearchBot. And for two answer-engine crawlers that both cite their sources, see DuckAssistBot vs PerplexityBot.