Bots
What is ClaudeBot, and should you allow it?
ClaudeBot is Anthropic's training crawler: Anthropic documents it as collecting web content that could contribute to training its Claude models. Blocking it keeps your future pages out of that training data and does not stop Claude finding or reading you, because search and user fetches use Claude-SearchBot and Claude-User. Allow it only if you accept training use.
7 days free · card required
68ClaudeBot requests to attensira.com
Estimated: 65% of Anthropic's 104 requests · #4 of 4 training agents
| Agent | Share of requests | Requests |
|---|---|---|
| Meta-ExternalAgent | 131,105 | |
| Bytespider | 3,505 | |
| GPTBot | ≈830 | |
| ClaudeBot (this page) | ≈68 |
≈: an operator's requests split by the purpose each of its agents declares. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP · · n=143,601 requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block ClaudeBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves ClaudeBot today.
# Decline the training corpus, keep Claude able to find# and read you.User-agent: ClaudeBotDisallow: / # Anthropic also supports the non-standard Crawl-delay# extension:# User-agent: ClaudeBot# Crawl-delay: 1User-agent: ClaudeBotAllow: /Disallow: /api/We name ClaudeBot and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs ClaudeBot, and what is it for?
Anthropic runs it, and we file it under training. Collects text that may enter a training corpus. Blocking it removes no page from any assistant's search index or citations, though a model can still name you from what it already learned.
From Anthropic’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Does blocking ClaudeBot remove you from Claude?
No. Anthropic runs three agents and gives each its own robots.txt token. ClaudeBot is the one Anthropic documents as collecting web content that could potentially contribute to training its generative AI models1. When you restrict it, Anthropic says, you signal that your future materials should be excluded from its training datasets3. That is the whole effect.
Claude finding your pages for a search is Claude-SearchBot's job, and Claude reading a page because someone asked is Claude-User's. A rule that names ClaudeBot touches neither, which is why the robots.txt block above declines training and says nothing else. The way sites lose Claude by accident is a Disallow: / under User-agent: *, which applies to every agent without a group of its own, all three of Anthropic's included.
Note the word future. A robots.txt rule written today says nothing about pages that were already collected.
How do I verify that a visit really came from ClaudeBot?
A user-agent header is free text, so a log line naming ClaudeBot is a claim until you check it.
Match the visiting IP against the published ranges in the record.
One combined file covers all three Anthropic agents, so a verified address proves the request is Anthropic's but not which of the three made it; the token still has to be read from the header.
Should you use Crawl-delay or an IP block instead?
If the problem is load rather than training, there is a gentler tool. Anthropic states it supports the non-standard Crawl-delay extension and gives a ClaudeBot group with a one-second delay as its example5. Support for it varies by operator, so it works only where the operator says it does, as Anthropic does here. For ClaudeBot it is documented, so throttling is a real option alongside blocking.
An IP block is the option Anthropic warns against. It states that blocking the addresses its bots operate from may not work correctly or persistently guarantee an opt-out, because it stops Anthropic reading your robots.txt6. That runs against the instinct of most operations teams. If you want out, say so in robots.txt and let the crawler read it.
Anthropic also states its bots honour robots.txt and do not bypass anti-circumvention technology such as CAPTCHAs4. That is a policy statement about Anthropic's crawler, not about the next request in your logs that claims to be ClaudeBot.
- IP range file
- bots.json
Anthropic publishes no reverse-DNS suffix for ClaudeBot; verify by the range file.
ResearcherTranscribed from Anthropic documentation · How often do sites block ClaudeBot?
Measured by us, not reported by Anthropic: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow ClaudeBot at the root
- 13.4%
- 1,077 of 8,036 domains
- allow it by name
- 3.4%
- 276 domains
- never mention it
- 83.2%
- 6,683 domains
Why is there no version to match?
Anthropic publishes the token ClaudeBot and robots.txt examples using it, but no full User-Agent header string2. Most operators publish a full string with a version number that drifts. Here there is nothing to pin. Match the token ClaudeBot, case-insensitively, as a whole word, and do not copy a "ClaudeBot user agent" from a third-party list: whatever version it shows is not from Anthropic's documentation.
Verification has one quirk. Anthropic publishes one combined address list for its three bots at claude.com/crawling/bots.json7. A match proves the request is Anthropic's. It does not prove which of the three agents made it; for that you still read the token from the header.
What does Anthropic's traffic look like on a real site?
Smaller than you might expect. On attensira.com, Attensira's own AI traffic measurement recorded 104 requests from Anthropic's agents across 48 pages in the 30 days to 4 October 2026, against 7,536 from OpenAI's in the same window. Two of Anthropic's three tokens appeared: ClaudeBot and Claude-User. The board read 35% of Anthropic's 104 requests as human-driven, a user-triggered fetch, a person's question in Claude reaching one of our pages.
The counts are user agents as sent and are not checked against claude.com/crawling/bots.json, so read them as direction. They are one site's numbers. One classification note: Attensira's AI traffic board currently files ClaudeBot's requests (65% of the 104) under search indexing, while Anthropic documents ClaudeBot as its training crawler. This page follows Anthropic's wording; the frame at the top splits the row by trigger, autonomous or human-driven, which needs no purpose label. The point is the ratio: on our site the training crawler and the fetcher a person triggers arrive under the same operator's name, and a single "block Anthropic" rule would have turned both away.
How is ClaudeBot different from GPTBot and Claude-User?
ClaudeBot
- Operator's documented job
- Collects content that could contribute to model training
- Blocking it
- Excludes future pages from Anthropic's training data
Claude-User
- Operator's documented job
- Fetches a page when a person asks Claude a question
- Blocking it
- May reduce visibility for user-directed search, per Anthropic
GPTBot
- Operator's documented job
- Crawls content that may be used for OpenAI's training
- Blocking it
- Same decision, different operator; no effect on ChatGPT search
The training-crawler pair across the two labs is set out in ClaudeBot vs GPTBot. The split inside Anthropic is in ClaudeBot vs Claude-User.
What can Attensira show you about ClaudeBot?
AI traffic, on the Growth and Business plans, counts which pages AI agents fetch, grouped by operator and by declared purpose. It is measured in edge middleware, because crawlers do not run JavaScript and a browser script never sees them. It reports user agents as sent and does not claim a verified identity per request. Whether Claude actually names you is a separate reading: prompt tracking reads Claude's answers to your buyers' questions every day. See crawler logs, or write the rule with the robots.txt generator.
ClaudeBotAgents filed beside ClaudeBot
Should I allow ClaudeBot in my situation?
Concrete cases, and what each one implies for this agent.
You object to your archive entering a training corpus but you want Claude to keep citing your documentation.
Disallow ClaudeBot by name. Say nothing about Claude-User or Claude-SearchBot.
Anthropic documents three agents with three jobs, and states that restricting ClaudeBot signals that the site's future materials should be excluded from its AI model training datasets. The other two are the ones that put your URL in front of a reader, and a rule naming ClaudeBot does not touch them.
ClaudeBot is not the problem in principle, but it is hammering a large archive faster than your origin likes.
Add Crawl-delay under the ClaudeBot group instead of disallowing it.
Anthropic states it supports the non-standard Crawl-delay extension and gives a ClaudeBot group with a one-second Crawl-delay as its own example. Blocking a crawler because of load is a policy decision taken for an operational reason, and this is the option that keeps the two separate.
Somebody proposes blocking Anthropic's addresses at the firewall to be certain.
Do not. Use robots.txt, and use the address list only to verify traffic.
Anthropic states that blocking the IP addresses its bots operate from may not work correctly or persistently guarantee an opt-out, because doing so impedes its ability to read your robots.txt file. A firewall rule can therefore defeat the very signal you were trying to send.
What else do people ask?
What is ClaudeBot?
Should I allow ClaudeBot?
Does ClaudeBot respect robots.txt?
Where do these facts come from?
7 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Anthropic documents ClaudeBot as helping enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training.
support.claude.com · retrieved
“helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training”
Anthropic's crawler documentation publishes the robots.txt token ClaudeBot and robots.txt examples using it, but publishes no full User-Agent header string for the agent.
support.claude.com · retrieved
Anthropic states that when a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from its AI model training datasets.
support.claude.com · retrieved
“it signals that the site's future materials should be excluded from our AI model training datasets”
Anthropic states that its bots respect "do not crawl" signals by honouring industry standard directives in robots.txt, and respect anti-circumvention technologies such as not bypassing CAPTCHAs.
support.claude.com · retrieved
“Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt”
Anthropic states it supports the non-standard Crawl-delay extension to robots.txt, giving a ClaudeBot group with a one-second Crawl-delay as its example.
support.claude.com · retrieved
“To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt”
Anthropic states that blocking the IP addresses its bots operate from may not work correctly or persistently guarantee an opt-out, as it impedes its ability to read your robots.txt file.
support.claude.com · retrieved
“blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out”
Anthropic publishes one combined address list covering its three bots at claude.com/crawling/bots.json, and states that a crawler whose source address is on that list is coming from Anthropic.
support.claude.com · retrieved
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Want whatever it takes to win the answer?
7 days free · card required