Bots
What is Claude-User, and should you allow it?
Claude-User is the agent Anthropic uses when a person asks Claude a question and Claude needs to read a web page. Allow it: Anthropic says blocking it may reduce your visibility for user-directed search. Unlike OpenAI's ChatGPT-User, it honours robots.txt, so a wildcard Disallow written for training crawlers really does turn it away.
7 days free · card required
36Claude-User requests to attensira.com
Estimated: 35% of Anthropic's 104 requests · #2 of 2 live answers agents
| Agent | Share of requests | Requests |
|---|---|---|
| ChatGPT-User | ≈230 | |
| Claude-User (this page) | ≈36 |
≈: an operator's requests split by the purpose each of its agents declares. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP · · n=143,601 requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Claude-User in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Claude-User today.
# Keep the fetch a person asked for. Decline the training# corpus.User-agent: Claude-UserAllow: / User-agent: ClaudeBotDisallow: /User-agent: Claude-UserAllow: /Disallow: /api/We name Claude-User and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs Claude-User, and what is it for?
Anthropic runs it, and we file it under user-triggered fetch. Reads one page when a person asks Claude something that needs it. Not a crawl and not training; Anthropic says blocking it may reduce your visibility for user-directed search.
From Anthropic’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Why should you allow Claude-User?
Because it only arrives when a person asked Claude something. Anthropic documents Claude-User as supporting Claude users: when individuals ask Claude a question, it may access websites with this agent1. One question, one fetch, one reader waiting on the answer.
Anthropic is direct about the cost of blocking it. Disabling Claude-User, it says, prevents its system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search3. That is an operator telling you, in its own documentation, that this rule costs you visibility.
The agent you may want to refuse is a different one. ClaudeBot is the token Anthropic documents for collecting content that could contribute to model training6. The robots.txt block above allows Claude-User and disallows ClaudeBot, which is what a site with a training objection and a visibility interest actually wants.
Does Claude-User follow robots.txt when ChatGPT-User may not?
Yes, and this is the difference most robots.txt files get wrong. Anthropic states that its bots respect "do not crawl" signals by honouring industry standard directives in robots.txt4. The statement covers its bots as a set, and Anthropic's own instructions repeat the block for Claude-User. There is no carve-out for the user-initiated agent.
OpenAI takes the opposite position for its equivalent, saying robots.txt rules may not apply to ChatGPT-User because a user initiated the action. Perplexity says its user fetcher generally ignores robots.txt. So a policy written once as "user-triggered AI fetchers are exempt" is wrong about Anthropic, and "they all obey it" is wrong about OpenAI.
The practical consequence runs one way. Because Claude-User honours robots.txt, a wildcard Disallow: / written to keep training crawlers out reaches it too. Claude then tells the person it could not read your page, and the answer gets built from other sources. If Claude has trouble opening your site, check your wildcard group first.
How do I verify that a visit really came from Claude-User?
A user-agent header is free text, so a log line naming Claude-User is a claim until you check it.
Match the visiting IP against the published ranges in the record.
The file is shared across all three Anthropic agents, so it verifies the operator but not which agent; the token must still be read from the header.
How is Claude-User different from ClaudeBot?
Claude-User
- Anthropic's documented job
- Accesses websites when a person asks Claude a question
- If you block it
- May reduce your visibility for user-directed search
ClaudeBot
- Anthropic's documented job
- Collects content that could contribute to model training
- If you block it
- Future pages are excluded from training data
Claude-SearchBot
- Anthropic's documented job
- Navigates the web to improve search result quality
- If you block it
- May reduce visibility and accuracy in search results
The full split is in ClaudeBot vs Claude-User. For how the other labs handle user-triggered fetches, see ChatGPT-User and PerplexityBot vs Perplexity-User.
How often does Claude-User show up on a real site?
On attensira.com, Attensira's own AI traffic measurement recorded 104 requests from Anthropic's agents across 48 pages in the 30 days to 4 October 2026. The board read 35% of them as human-driven, about 36 requests, each a user-triggered fetch: a person's question in Claude that reached one of our pages. The rest came from ClaudeBot. That share is far higher than OpenAI's on the same site, where live-answer fetches were 3% of 7,536 requests.
These are user agents as sent, not checked against Anthropic's address list, and one site's mix. But they show why the distinction matters: each Claude-User request is a reader, and a wildcard rule aimed at ClaudeBot would have refused them all.
- IP range file
- bots.json
Anthropic publishes no reverse-DNS suffix for Claude-User; verify by the range file.
ResearcherTranscribed from Anthropic documentation · Agents filed beside Claude-User
How often do sites block Claude-User?
Measured by us, not reported by Anthropic: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow Claude-User at the root
- 5.5%
- 438 of 8,036 domains
- allow it by name
- 1.7%
- 134 domains
- never mention it
- 92.9%
- 7,464 domains
Why is there no version string to match?
Anthropic publishes the token Claude-User but no full User-Agent header string2. There is no version to drift and nothing to pin. Match the token Claude-User, case-insensitively, as a whole word. Verification uses one combined list for all three Anthropic agents, at claude.com/crawling/bots.json5. It proves the request is Anthropic's; the token tells you which agent.
What can Attensira show you about Claude-User?
AI traffic, on Growth and Business, counts which pages AI agents fetch, grouped by operator and declared purpose, so user-triggered fetches are counted apart from crawls. It is measured at the edge, since none of these agents runs JavaScript, and it reports user agents as sent rather than claiming a verified identity. Prompt tracking reads Claude's answers to your buyers' questions every day, which is where those fetches end up. See crawler logs and the robots.txt generator.
Claude-UserShould I allow Claude-User in my situation?
Concrete cases, and what each one implies for this agent.
A blanket AI block went in last quarter and support is now fielding reports that Claude cannot read your help centre when customers paste links.
Allow Claude-User explicitly, and leave the ClaudeBot rule where it is.
Anthropic states that disabling Claude-User prevents its system from retrieving your content in response to a user query. Those requests correspond one-to-one with a person asking about you, so the block is turning away readers while doing nothing about training.
You need a defensible written policy that no Anthropic agent reads a specific directory.
Write the robots.txt rule, and note that Anthropic states its bots honour robots.txt rather than relying on the assumption that user-initiated fetches are exempt.
Anthropic's robots.txt statement is made about its bots as a set, with no exemption for the user-initiated agent. That is stronger than the position OpenAI takes for its equivalent token, which is why an "AI fetchers ignore robots.txt" policy written once for every operator will be wrong about at least one of them.
Your dashboard shows Anthropic traffic rising and you want to know whether it is training or reading.
Verify against the shared address list, then split the verified traffic by token.
Anthropic publishes one combined file for all three agents, so verification proves only that the request is Anthropic's. The training-versus-citation split lives entirely in the token, and a report that stops at "Anthropic traffic up 40 percent" has not answered the question anyone was asking.
What else do people ask?
What is Claude-User?
Does Claude-User respect robots.txt?
Why does Claude say it cannot open my page?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Anthropic documents Claude-User as supporting Claude AI users, accessing websites when individuals ask questions to Claude.
support.claude.com · retrieved
“Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent”
Anthropic's crawler documentation publishes the robots.txt token Claude-User but publishes no full User-Agent header string for the agent.
support.claude.com · retrieved
Anthropic states that disabling Claude-User prevents its system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
support.claude.com · retrieved
“prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search”
Anthropic states that its bots respect "do not crawl" signals by honouring industry standard directives in robots.txt, making the statement about its bots as a set.
support.claude.com · retrieved
“Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt”
Anthropic publishes one combined address list covering its three bots at claude.com/crawling/bots.json, and states that a crawler whose source address is on that list is coming from Anthropic.
support.claude.com · retrieved
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Anthropic documents ClaudeBot, a separate robots.txt token, as collecting web content that could potentially contribute to the training of its generative AI models.
support.claude.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required