ClaudeBot vs Claude-User: Anthropic's training crawler and its live fetcher

What do ClaudeBot and Claude-User actually do?

ClaudeBot is Anthropic's crawler for model development. Anthropic says it helps enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training, and that restricting it signals your site's future materials should be excluded from its training datasets. It crawls on Anthropic's schedule, not because anyone asked about you.

Claude-User acts for a person. Anthropic says that when individuals ask Claude questions, it may access websites using a Claude-User agent, and that disabling it prevents Anthropic's system retrieving your content in response to a user query. It fires during a conversation: someone asks Claude about your category, Claude decides your page is worth reading, and Claude-User fetches it.

The two have nothing in common except the operator. One builds future models. The other serves a present answer. Treating them as one decision, which a blanket "block Anthropic" rule does, gives up the second to stop the first.

What should your robots.txt say?

For a site that wants to stay out of training but be found and read by Claude:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

If your concern is load rather than reuse, Anthropic supports the non-standard Crawl-delay extension and shows it under ClaudeBot. A delay slows the training crawler without excluding anything:

User-agent: ClaudeBot
Crawl-delay: 5

Anthropic adds that the rules must sit on every subdomain you want covered. A blog or docs host with its own robots.txt needs the same three groups.

How do large sites treat the three Anthropic tokens?

On 3 September 2026 we read the robots.txt of the top 10,000 domains in Tranco list K9QPW and parsed 8,036. 1,077 (13.4%) disallow ClaudeBot at the root. Claude-User and Claude-SearchBot are each disallowed by about 5.5% (438 and 442). And 7,464 (92.9%) never mention Claude-User at all, so their wildcard group decides for them. Large sites are, on the whole, already making the split this page recommends: training out more often, live fetching and search in.

The roughly 440 sites blocking Claude-User are worth a thought. Because Anthropic honours that rule, those sites have chosen to be unreadable to Claude at the moment a person asks about them. Some will have meant it. Others will have copied a list of AI user agents into one disallow group without asking what each one does. The full table is in our study of AI crawler access.

What did Anthropic's agents do on our site?

Attensira counts crawler requests at the edge on attensira.com, as it does for customers on Growth and Business. In the 30 days to 4 October 2026, 104 requests carried Anthropic user agents, across 48 pages. About 65% were autonomous crawls, ClaudeBot's pattern. About 35% were fetched because a person asked Claude something, which is Claude-User. No request was labelled Claude-SearchBot in the window.

Two things stand out. Anthropic's total is small next to OpenAI's: the same site logged 7,536 OpenAI requests in the same period. And the person-triggered share is high: about a third of Anthropic's requests were people asking Claude something that needed one of our pages, against about 3% for OpenAI's live fetcher on our site. A share of 104 moves a lot with a few requests, and every label is the user agent the request claimed, with none IP-verified, so we read it as a hint rather than a finding. But a hint that much of Claude's contact with a site happens live, at answer time, is a reason not to block Claude-User casually.

Why does the live fetch matter more than the crawl?

Because it is the step closest to a buyer. A training crawl affects what some future model might know about your category, through a path nobody can trace for a single site. A live fetch affects what one person is told today, by a model that decided your page was worth reading for their question.

There is a freshness argument too. A model's training data is months old by the time anyone uses it. A live fetch reads your page as it is now: the current price, the current feature list. If a buyer asks Claude about you after you changed your pricing, Claude-User is how the answer can reflect the change. Blocking it means Claude answers from whatever else it has, which may be older, and may be somebody else's description of you.

Does any of this change what Claude says about you?

Only at the edges. Allowing Claude-User and Claude-SearchBot makes your pages readable and findable; it does not make Claude choose them. What an assistant says about your category depends on which sources it trusts for the question, and for most commercial questions those are review sites, comparison pages, documentation and trade press as much as your own site. Access is the floor. The work above the floor is publishing pages that answer the question better than the pages the assistant currently reads, and earning mentions on the pages it already trusts.

Our own readings show how far apart the two are. In the same 30 days that Anthropic's and OpenAI's agents read our pages, neither ChatGPT nor Perplexity named or cited Attensira in any of 60 answers to "list of AI crawler user agents". Being fetched is not being chosen.