Bots
Amzn-SearchBot: allow it or block it?
Amzn-SearchBot is Amazon's search crawler: allowing it makes your content eligible to appear in Amazon search experiences such as Alexa, and Amazon states it does not crawl for generative AI model training. Allow it. If you say nothing about it in robots.txt, Amazon says it follows the rules you gave other search bots, so write an explicit group.
7 days free · card required
0requests filed as Amzn-SearchBot · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| Amzn-SearchBot (this page) | 0 | Parent's rule |
| OAI-SearchBot | ≈6,480 | Allowed |
| PerplexityBot | 621 | Allowed |
| Claude-SearchBot | 0 | Allowed |
| meta-webindexer | 0 | Wildcard |
| MistralAI-Index | 0 | Wildcard |
| YouBot | 0 | Allowed |
2 of 7 had requests filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Amzn-SearchBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Amzn-SearchBot today.
# Stay eligible for Amazon search experiences such as# Alexa.User-agent: Amzn-SearchBotAllow: /# No group names Amzn-SearchBot, so it follows our rules# for other search bots, such as:User-agent: GooglebotAllow: /Disallow: /api/We do not name Amzn-SearchBot. Amazon says it follows the rules given to other search bots, and every search bot our file names, Googlebot among them, is allowed everywhere except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs Amzn-SearchBot, and what is it for?
Amazon runs it, and we file it under search indexing. Builds the index an assistant searches when it answers. Blocking it removes you from the pool a citation is drawn from.
From Amazon’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Why should you allow Amzn-SearchBot?
Because it is the Amazon agent that can surface your page to a person, and Amazon says it does not train. Amazon documents Amzn-SearchBot as used to improve search experiences in Amazon products and services, and adds the sentence that settles the question: by permitting Amzn-SearchBot access, your content is eligible to appear in search experiences such as Alexa1.
In the same table, Amazon states that Amzn-SearchBot does not crawl content for generative AI model training, and that its three user-agent settings are independent of one another2. That makes the usual position easy to express: allow Amzn-SearchBot, and if you object to training, disallow Amazonbot, the Amazon agent whose output may be used to train Amazon AI models.
What does Amzn-SearchBot do when robots.txt never mentions it?
It borrows rules you wrote for other crawlers. Amazon states that if robots.txt files do not mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the directives given to those other search bots3.
Most crawlers treat a missing group as missing and fall through to User-agent: *. This one is documented as inheriting what you told other search bots. If your file has permissive groups for other search bots and a restrictive wildcard, Amazon's wording points this token at the search-bot rules, not the wildcard, which is probably not the one you would have guessed. A blanket AI blocklist has the opposite failure: it often sweeps Amzn-SearchBot in alongside Amazonbot.
The fix is one line of intent. The robots.txt block above names Amzn-SearchBot with an explicit Allow, which removes the inference. Amazon states these agents honour user-agent and allow/disallow directives and do not support crawl-delay5.
How do I verify that a visit really came from Amzn-SearchBot?
A user-agent header is free text, so a log line naming Amzn-SearchBot is a claim until you check it.
Match the visiting IP against the published ranges in the record.
Amazon keeps a dedicated list per token. Verify against the list that matches the token in the header, and when the header and the list disagree, believe the address: it identifies which Amazon agent called.
How is Amzn-SearchBot different from OAI-SearchBot?
They are the same kind of agent at two companies: a search crawler with a separately named training crawler beside it.
What it feeds
- Amzn-SearchBot (Amazon)
- Amazon search experiences such as Alexa
- OAI-SearchBot (OpenAI)
- ChatGPT's search results7
Training
- Amzn-SearchBot (Amazon)
- Amazon says it does not crawl for model training
- OAI-SearchBot (OpenAI)
- Training is a separate token, GPTBot9
Training token beside it
- Amzn-SearchBot (Amazon)
- Amazonbot
- OAI-SearchBot (OpenAI)
- GPTBot
If robots.txt is silent about it
- Amzn-SearchBot (Amazon)
- Follows rules given to other search bots
- OAI-SearchBot (OpenAI)
- Not documented
What blocking costs
- Amzn-SearchBot (Amazon)
- Eligibility for Amazon's search experiences
- OAI-SearchBot (OpenAI)
- Not shown in ChatGPT search answers; navigational links may remain8
Address list
- Amzn-SearchBot (Amazon)
- Dedicated per-token list
- OAI-SearchBot (OpenAI)
- Dedicated per-agent file
The shared lesson is that "the Amazon bot" and "the OpenAI bot" are not single things. A rule written about training that names a search token is in the wrong place. OpenAI's side has its own page at OAI-SearchBot, and the search-crawler pair across OpenAI and Perplexity is in OAI-SearchBot vs PerplexityBot.
Why might you allow it and still see no traffic?
Three reasons, in order of likelihood. Amazon may use a cached copy of robots.txt, so a new rule takes time. A WAF or CDN bot rule may refuse an unfamiliar header. Or Amazon simply has not needed your pages yet. Allowing a crawler is not the same as being crawled, and being crawled is not the same as being cited.
- IP range file
- searchbot-ip-addresses
Amazon publishes no reverse-DNS suffix for Amzn-SearchBot; verify by the range file.
ResearcherTranscribed from Amazon documentation · Agents filed beside Amzn-SearchBot
How should you match it in logs?
Amazon's example header is Amzn-SearchBot/0.14, with the Chrome version written as a placeholder. Match the bare token Amzn-SearchBot, case-insensitively, as a whole word. It shares no prefix with Amazonbot, so a filter looking for "amazon" misses it entirely, which is how sites conclude Amazon stopped crawling them.
What can Attensira show you about Amzn-SearchBot?
AI traffic, on Growth and Business, counts crawler fetches per operator and declared purpose at the edge, so an allow-search, block-training split for Amazon becomes something you can observe. It reports user agents as sent; the address check stays yours. See crawler logs and the robots.txt generator.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36Should I allow Amzn-SearchBot in my situation?
Concrete cases, and what each one implies for this agent.
You wrote a blanket AI blocklist and Amzn-SearchBot went into it alongside Amazonbot.
Split them — allow Amzn-SearchBot, keep the Amazonbot rule if training is your objection.
Amazon documents these as independent settings with different jobs, and states that Amzn-SearchBot does not crawl content for generative AI model training. Blocking it forfeits eligibility for Amazon search experiences without changing anything about training.
Your robots.txt has rules for Googlebot and Bingbot and says nothing about Amazon.
Decide deliberately, because Amazon has documented what silence means here.
Amazon states that if a robots.txt file does not mention Amzn-SearchBot but allows other search bots, Amzn-SearchBot will crawl in accordance with the directives given to other search bots. Silence is not neutral for this token — it inherits whatever you told the other search crawlers.
You allowed the token but see no Amzn-SearchBot traffic, and suspect a WAF is refusing it.
Allowlist the searchbot address list at the edge, then re-check your logs against that list.
robots.txt only governs a crawler that reached your origin. If the connection is refused earlier, the permissive rule is invisible to Amazon. Verifying against the dedicated address list also tells you whether the traffic you do see is genuinely this token rather than a header borrowing the name.
What else do people ask?
What is Amzn-SearchBot?
What happens if my robots.txt does not mention Amzn-SearchBot?
Does Amzn-SearchBot support crawl-delay?
Where do these facts come from?
9 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Amazon states that Amzn-SearchBot is used to improve search experiences in Amazon products and services, and that by permitting it access, a site's content is eligible to appear in search experiences such as Alexa.
developer.amazon.com · retrieved
“Amzn-SearchBot is used to improve search experiences in Amazon products and services. By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa.”
Amazon states that Amzn-SearchBot does not crawl content for generative AI model training, and that each of its three documented user-agent settings is independent of the others.
developer.amazon.com · retrieved
“Amzn-SearchBot does not crawl content for generative AI model training.”
Amazon states that if robots.txt files do not mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots.
developer.amazon.com · retrieved
“If robots.txt files don't mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots.”
Amazon gives Amzn-SearchBot's example user-agent string as "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36".
developer.amazon.com · retrieved · changes often, check the source
Amazon states that automated crawling from its listed user agents, which include Amzn-SearchBot, respects the Robots Exclusion Protocol, honouring the user-agent and allow/disallow directives, and that they do not support crawl-delay.
developer.amazon.com · retrieved
Amazon publishes Amzn-SearchBot's addresses at developer.amazon.com/amazonbot/searchbot-ip-addresses/, separately from the Amazonbot and Amzn-User lists.
developer.amazon.com · retrieved
OpenAI documents OAI-SearchBot as the agent used to surface websites in search results in ChatGPT's search features.
developers.openai.com · retrieved
OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
developers.openai.com · retrieved
OpenAI documents GPTBot, a separate robots.txt token, as the agent used to crawl content that may be used in training its generative AI foundation models.
developers.openai.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required