Bots
AI search crawlers
AI search bots build the indexes AI assistants search when they answer: OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot, meta-webindexer for Meta AI, MistralAI-Index, Amzn-SearchBot and YouBot. Blocking one takes your pages out of the pool that assistant cites from. Their operators say they respect robots.txt, so allow them by name.
7 days free · card required
- agents documented
- 7
- publish IP ranges or reverse DNS
- 6 of 7
- requests to attensira.com, 30 days
- ≈7,100
- of n=143,601 AI crawler requests
- of top sites disallow PerplexityBot, the most blocked of the 3 in the study
- 8.2%
- n=8,036 robots.txt files
One of the category's rules, from the OAI-SearchBot entry, per the operator's documentation; the other 6 are on their own entries.
User-agent: OAI-SearchBotAllow: /
ResearcherOperator documentation, our AI traffic board (Attensira MCP) and robots.txt study · Which AI search crawlers are documented?
Builds the index an assistant searches when it answers. Blocking it removes you from the pool a citation is drawn from.
- Amzn-SearchBot Amzn-SearchBot makes your pages eligible for Amazon search experiences such as Alexa, and does not crawl for AI training. How it compares with OAI-SearchBot. Amazon · Search indexing
- Claude-SearchBot Claude-SearchBot indexes pages for Claude's search answers. How it differs from ClaudeBot and OAI-SearchBot, and why most sites should allow it. Anthropic · Search indexing
- meta-webindexer Meta says allowing meta-webindexer helps Meta AI cite and link your content. Why it is the Meta crawler to keep, the robots.txt rule, and its limits. Meta · Search indexing
- MistralAI-Index MistralAI-Index indexes pages for Mistral's assistant search and, per Mistral, never feeds training. The robots.txt rule and the IP file that verifies it. Mistral AI · Search indexing
- OAI-SearchBot OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers. The robots.txt rule, the IP file, and how attensira.com configures it. OpenAI · Search indexing
- PerplexityBot PerplexityBot builds the index behind Perplexity's cited answers and, per Perplexity, never trains models. The rule, the IP file, and what it did on our site. Perplexity · Search indexing
- YouBot YouBot indexes for You.com search and is documented with three ways to verify a visit, including signed requests. The robots.txt rule and the checks. You.com · Search indexing
Which AI search crawlers fetched attensira.com in the last 30 days?
2 of 7 had requests filed on attensira.com in the window. User agents as sent, not checked against address lists.
| Agent | Requests | Our robots.txt |
|---|---|---|
| OAI-SearchBot | ≈6,480 | Allowed by name |
| PerplexityBot | 621 | Allowed by name |
| Amzn-SearchBot | 0 | Follows other search bots |
| Claude-SearchBot | 0 | Allowed by name |
| meta-webindexer | 0 | No named group |
| MistralAI-Index | 0 | No named group |
| YouBot | 0 | Allowed by name |
How do the AI search crawlers compare?
Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.
Sorted by top sites blocking, high to low.
| Agent | |||||
|---|---|---|---|---|---|
| PerplexityBot | Perplexity | Respected | IP ranges | Allowed by name | 8.2% |
| OAI-SearchBot | OpenAI | Respected | IP ranges | Allowed by name | 5.8% |
| Claude-SearchBot | Anthropic | Respected | IP ranges | Allowed by name | 5.5% |
| Amzn-SearchBot | Amazon | Respected | IP ranges | Follows other search bots | Not in study |
| meta-webindexer | Meta | Respected | Nothing published | No named group | Not in study |
| MistralAI-Index | Mistral AI | Respected | IP ranges | No named group | Not in study |
| YouBot | You.com | Respected | reverse DNS | Allowed by name | Not in study |
PerplexityBot
- Operator
- Perplexity
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Allowed by name
- Top sites blocking
- 8.2%
OAI-SearchBot
- Operator
- OpenAI
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Allowed by name
- Top sites blocking
- 5.8%
Claude-SearchBot
- Operator
- Anthropic
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Allowed by name
- Top sites blocking
- 5.5%
Amzn-SearchBot
- Operator
- Amazon
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Follows other search bots
- Top sites blocking
- Not in study
meta-webindexer
- Operator
- Meta
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- No named group
- Top sites blocking
- Not in study
MistralAI-Index
- Operator
- Mistral AI
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- No named group
- Top sites blocking
- Not in study
YouBot
- Operator
- You.com
- robots.txt
- Respected
- Verify by
- reverse DNS
- attensira.com
- Allowed by name
- Top sites blocking
- Not in study
Each row read from the operator's documentation, 3 Sep 2026.
“Not in study” means the token is not one our robots.txt study counts yet, not that no site blocks it.
Why are these the bots that decide AI citations?
An assistant that searches before it answers searches an index of its own, and the search crawler is how a page gets into it. OpenAI documents OAI-SearchBot as surfacing websites in ChatGPT's search features. Meta says allowing meta-webindexer helps Meta AI cite and link your content. Anthropic says blocking Claude-SearchBot reduces your visibility in Claude's search results. A page that is not in the index cannot be the citation.
- IP range file
- bots.json
Anthropic publishes no reverse-DNS suffix for Claude-SearchBot; verify by the range file.
One file covers all three Anthropic agents, so it verifies the operator and the header token identifies the agent. The Claude-SearchBot entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Anthropic documentation · How do I allow AI search but refuse training?
Name each search token in its own Allow group and each training token in its own Disallow group. The operators split them: OAI-SearchBot and GPTBot, Claude-SearchBot and ClaudeBot, meta-webindexer and meta-externalagent, MistralAI-Index and MistralAI-Training, Amzn-SearchBot and Amazonbot.
Avoid a wildcard Disallow aimed at AI companies. An agent with no group of its own falls back to the * group, so a wildcard Disallow reaches every search crawler you did not name. Mistral says MistralAI-Index content is not used for training, Amazon says the same of Amzn-SearchBot, and Perplexity says PerplexityBot is not used for foundation-model training.
Meta publishes no reverse-DNS suffix and no IP range file for meta-webindexer, so a visit can be matched only by its user agent.
The meta-webindexer entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Meta documentation · Does allowing them get me cited?
It makes a page eligible, nothing more. Whether an answer names you depends on what the page says and what else the engine found. That is what Attensira measures: it reads the answers your buyers' questions get every day and shows which pages each engine cites, then its agents do the work to close the gaps.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https:/The version number in this string changes; match the token, not the whole string. The OAI-SearchBot entry, and reading user-agent strings.
ResearcherTranscribed from OpenAI documentation · How often do the most visited sites block AI search crawlers?
| Name | Disallowed at the root | Count |
|---|---|---|
| PerplexityBot | 8.2% | 655 |
| OAI-SearchBot | 5.8% | 467 |
| Claude-SearchBot | 5.5% | 442 |
Web engineer · Writes robots rules like theseRobots.txt access study, our own fetch · · n=8,036 robots.txt filesPerplexityBot is the most blocked here: 655 of 8,036 sites disallow it at the root. The least blocked, Claude-SearchBot, is disallowed by 442.
A site that never names an agent is counted as unspecified, not as allowing it.
What else do people ask?
What is an AI search crawler?
Does blocking OAI-SearchBot remove me from ChatGPT?
Do AI search crawlers train models on my content?
Can I verify an AI search crawler visit?
Where do these facts come from?
8 claims on this page, each with the page it was read from; last checked .
Amazon states that Amzn-SearchBot is used to improve search experiences in Amazon products and services, and that by permitting it access, a site's content is eligible to appear in search experiences such as Alexa.
developer.amazon.com · retrieved
“Amzn-SearchBot is used to improve search experiences in Amazon products and services. By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa.”
OpenAI documents OAI-SearchBot as the agent used to surface websites in search results in ChatGPT's search features.
developers.openai.com · retrieved
Anthropic documents Claude-SearchBot as navigating the web to improve search result quality for users, analysing online content to enhance the relevance and accuracy of search responses.
support.claude.com · retrieved
“Claude-SearchBot navigates the web to improve search result quality for users”
Meta states that the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users, that Meta analyses online content to enhance the relevance and accuracy of Meta AI, and that allowing Meta-WebIndexer in robots.txt helps Meta cite and link to a site's content in Meta AI's responses.
developers.facebook.com · retrieved
“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users.”
RFC 9309 section 2.2.1 states that crawlers obey the groups whose user-agent line matches their product token, combined if there are several, and obey the group with the "*" user-agent only if no matching group exists.
rfc-editor.org · retrieved
“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”
Mistral states that MistralAI-Index is for automated crawling of the web for indexing purposes only, and that it indexes content for Mistral search, which helps answer user questions in Mistral's assistant.
docs.mistral.ai · retrieved
“MistralAI-Index is for automated crawling of the web for indexing purposes only.”
Perplexity documents PerplexityBot as designed to surface and link websites in search results on Perplexity, and states it is not used to crawl content for AI foundation models.
docs.perplexity.ai · retrieved
“PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.”
You.com states that YouBot is the web crawler that powers the You.com search engine, automatically discovering and indexing web pages to provide real-time, accurate search results for You.com users.
you.com · retrieved
“YouBot is the web crawler that powers the You.com search engine. It automatically discovers and indexes web pages to provide real-time, accurate search results for You.com users.”
Want whatever it takes to win the answer?
7 days free · card required