Bots
Claude-SearchBot: allow it or block it?
Claude-SearchBot is Anthropic's search crawler: it reads the web to improve the quality of Claude's search results, not to train models. Allow it if you want Claude to describe you accurately. Anthropic says blocking it may reduce your visibility and accuracy in search results. To refuse training, block ClaudeBot instead; the two tokens are independent.
7 days free · card required
0requests filed as Claude-SearchBot
Anthropic sent 104, all from ClaudeBot, Claude-User
| Agent | Share of requests | Requests |
|---|---|---|
| ClaudeBot | ≈68 | |
| Claude-User | ≈36 | |
| Claude-SearchBot (this page) | 0 |
≈: an operator's requests split by the purpose each of its agents declares. No request in the window was filed as Claude-SearchBot. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP · · n=143,601 requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Claude-SearchBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Claude-SearchBot today.
# Stay indexed for Claude's search. Decline the training# corpus.User-agent: Claude-SearchBotAllow: / User-agent: ClaudeBotDisallow: /User-agent: Claude-SearchBotAllow: /Disallow: /api/We name Claude-SearchBot and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs Claude-SearchBot, and what is it for?
Anthropic runs it, and we file it under search indexing. Builds the index an assistant searches when it answers. Blocking it removes you from the pool a citation is drawn from.
From Anthropic’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
What does Claude-SearchBot add that Claude does not already have?
Currency, which is why you should let it in if you want Claude to describe you accurately. Anthropic documents Claude-SearchBot as navigating the web to improve search result quality for users, analysing content to make search responses more relevant and accurate1. It is the crawler that keeps Claude's picture of your pages current.
Anthropic also says what blocking it costs. Disabling Claude-SearchBot prevents its system from indexing your content for search optimisation, which may reduce your site's visibility and accuracy in user search results3. Read the word accuracy closely. Blocking a search crawler does not make Claude stop talking about you. Models know about you from other sources. What this crawler adds is currency: the price you changed last month, the product you renamed, the feature you retired. Refuse it and Claude keeps answering with whatever it already has, which is often worse than silence, because a wrong description is harder to notice than an absent one.
Is Claude-SearchBot the same as ClaudeBot?
No. They are separate tokens with separate jobs. ClaudeBot is the agent Anthropic documents as collecting content that could potentially contribute to training its models6. Claude-SearchBot is for search. The block in the robots.txt section above allows Claude-SearchBot and disallows ClaudeBot, which is the configuration for a site that wants to be found but objects to training.
Anthropic states that its bots honour robots.txt and tells site owners to repeat the block for Claude-SearchBot as needed4. That cuts both ways. A wildcard Disallow: / written to keep training crawlers out reaches this crawler too, so give each Anthropic token its own group rather than relying on one rule for "the Anthropic bot".
The test is simple. Read your robots.txt and ask, for every AI-company token in it, which question that line was answering. If a line was written about training and it names a search crawler, it is in the wrong place.
How do I verify that a visit really came from Claude-SearchBot?
A user-agent header is free text, so a log line naming Claude-SearchBot is a claim until you check it.
Match the visiting IP against the published ranges in the record.
One file covers all three Anthropic agents, so it verifies the operator and the header token identifies the agent.
How is Claude-SearchBot different from OAI-SearchBot?
They are the two search crawlers that stand between your pages and the answers in ChatGPT and Claude, and both operators say blocking them has a cost. The cost is worded differently.
Documented job
- Claude-SearchBot (Anthropic)
- Improve search result quality for Claude users
- OAI-SearchBot (OpenAI)
- Surface websites in ChatGPT's search results7
What blocking costs
- Claude-SearchBot (Anthropic)
- May reduce visibility and accuracy in search results
- OAI-SearchBot (OpenAI)
- Not shown in ChatGPT search answers, though navigational links may remain8
Published user agent
- Claude-SearchBot (Anthropic)
- Token only, no full string
- OAI-SearchBot (OpenAI)
- Full string with a version number
Address list
- Claude-SearchBot (Anthropic)
- One file for all three Anthropic agents
- OAI-SearchBot (OpenAI)
- A file for the search crawler alone9
Anthropic describes a degradation: the picture of you gets blurrier. OpenAI describes an exclusion: the picture no longer contains you. A bare navigational link is not a citation and does not carry your framing, so it is not a loophole worth planning around.
The verification difference matters if you need to prove something to a colleague. OpenAI's per-agent file proves on its own that a request came from its search crawler. With Anthropic, the address proves the operator and the token tells you the agent, so a log query separating Anthropic's search traffic from its training traffic is trusting a header. The OpenAI side has its own page at OAI-SearchBot, and the search-crawler pair across OpenAI and Perplexity is in OAI-SearchBot vs PerplexityBot.
Does allowing Claude-SearchBot get you cited?
It is necessary and nowhere near sufficient. Access puts your pages in the pool Claude's search draws from. Whether a page is quoted depends on whether it answers a real question, in a form that can be lifted, better than the pages being quoted now. No robots.txt line touches that.
On attensira.com, our own AI traffic measurement recorded 104 requests from Anthropic's agents in the 30 days to 4 October 2026, and none of them identified itself as Claude-SearchBot; the traffic was ClaudeBot and Claude-User. One site and one window, with user agents as sent, but a useful reminder that a crawler being allowed is not the same as a crawler arriving.
- IP range file
- bots.json
Anthropic publishes no reverse-DNS suffix for Claude-SearchBot; verify by the range file.
ResearcherTranscribed from Anthropic documentation · Agents filed beside Claude-SearchBot
In this entry
How often do sites block Claude-SearchBot?
Measured by us, not reported by Anthropic: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow Claude-SearchBot at the root
- 5.5%
- 442 of 8,036 domains
- allow it by name
- 1.7%
- 137 domains
- never mention it
- 92.8%
- 7,457 domains
Why is there no version string to match?
Anthropic publishes the token Claude-SearchBot but no full User-Agent header string2. Match the token, case-insensitively, as a whole word. Verify against the shared address list at claude.com/crawling/bots.json5.
What can Attensira show you about Claude-SearchBot?
Two different readings. AI traffic, on Growth and Business, shows which pages AI agents fetch, by operator and declared purpose, measured at the edge and reported as the user agents sent. Prompt tracking reads what Claude actually answers to your buyers' questions every day, and the crew works on the pages those answers lean on. A crawl is an input; the answer is the result. See crawler logs, or build the rule with the robots.txt generator.
Claude-SearchBotShould I allow Claude-SearchBot in my situation?
Concrete cases, and what each one implies for this agent.
You blocked ClaudeBot last year using a rule that also names every token starting with Claude, and Claude has stopped surfacing your pages.
Split the rule into per-token groups and allow Claude-SearchBot explicitly.
Anthropic states that disabling Claude-SearchBot prevents indexing of your content for search optimisation and may reduce your visibility in user search results. A pattern-style rule that swept up the search crawler with the training crawler produces exactly that outcome, with no error anywhere.
A stakeholder asks for the Claude-SearchBot user-agent header so the CDN team can write a rule.
Give them the token, and tell them Anthropic does not publish a header string.
Handing over a header string sourced from a bot directory looks helpful and creates a rule pinned to something no operator ever committed to. The token is what robots.txt is addressed to and what a case-insensitive whole-word log filter should match, and it is what Anthropic actually published.
You need to report how much of your Anthropic traffic is citation-capable rather than training.
Verify addresses against the shared file, then split the verified requests by token.
Anthropic publishes one address list for all three agents, so verification alone cannot separate them. Claude-SearchBot and Claude-User are the citation-capable side, ClaudeBot is not, and only the header token carries that distinction once the address check has passed.
What else do people ask?
What is Claude-SearchBot?
Is Claude-SearchBot used to train Claude?
What is the difference between Claude-SearchBot and OAI-SearchBot?
Where do these facts come from?
9 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Anthropic documents Claude-SearchBot as navigating the web to improve search result quality for users, analysing online content to enhance the relevance and accuracy of search responses.
support.claude.com · retrieved
“Claude-SearchBot navigates the web to improve search result quality for users”
Anthropic's crawler documentation publishes the robots.txt token Claude-SearchBot but publishes no full User-Agent header string for the agent.
support.claude.com · retrieved
Anthropic states that disabling Claude-SearchBot prevents its system from indexing your content for search optimisation, which may reduce your site's visibility and accuracy in user search results.
support.claude.com · retrieved
“prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results”
Anthropic states that its bots respect "do not crawl" signals by honouring industry standard directives in robots.txt, and instructs site owners to repeat the block for Claude-SearchBot as needed.
support.claude.com · retrieved
“Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt”
Anthropic publishes one combined address list covering its three bots at claude.com/crawling/bots.json, and states that a crawler whose source address is on that list is coming from Anthropic.
support.claude.com · retrieved
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Anthropic documents ClaudeBot, a separate robots.txt token, as collecting web content that could potentially contribute to the training of its generative AI models.
support.claude.com · retrieved
OpenAI documents OAI-SearchBot as the agent used to surface websites in search results in ChatGPT's search features.
developers.openai.com · retrieved
“surface websites in search results in ChatGPT's search features”
OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
developers.openai.com · retrieved
OpenAI publishes the addresses OAI-SearchBot fetches from in a crawler-specific file at openai.com/searchbot.json.
developers.openai.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required