Anthropic

What does Anthropic fetch from a website?

Very little, and not the way most people guess. The strip at the top is attensira.com's crawler log for the 30 days to 4 October 2026, read over the Attensira MCP: 104 requests from Anthropic's agents, on 48 pages, out of 143,601 requests from AI crawlers of every kind.

About 35% of those requests were Claude-User: a person asked Claude something, and Claude fetched one of our pages to answer. That is the closest thing a log can show to "Claude read you because someone asked". The other 65% were autonomous requests from ClaudeBot. Two caveats sit on those numbers. The split is derived from a rounded purpose mix, so "about 68" and "about 36" are estimates, not counts. And the log classes ClaudeBot's requests as search indexing, while Anthropic documents ClaudeBot as its training crawler; treat it as both. We saw no Claude-SearchBot requests in the window, and none of the 104 was IP-verified.

Two practical points follow. Most of our site has not been fetched by Claude recently, and a page Claude has not fetched cannot be quoted from a fresh read. And the user-triggered share is real: blocking Anthropic broadly would have refused those people their answer.

What is Anthropic?

Anthropic is an AI safety and research company that builds the Claude family of large language models and the Claude assistant. Its company page describes its aim as building AI systems that are reliable, interpretable and steerable, and states that it is a public benefit corporation with a Long-Term Benefit Trust alongside its board.

For marketers and site owners, Anthropic matters in three ways: Claude answers questions people ask about your category, Anthropic's crawlers decide what Claude can read from your site, and the Model Context Protocol, which Anthropic introduced, is how many AI tools now connect to outside data. See Claude and Model Context Protocol.

Which crawlers does Anthropic run?

Three, documented separately in Anthropic's support article on crawling:

• ClaudeBot

  Documented job

  collects web content that could contribute to training

  If you block it

  future training excludes your content; answers can still fetch you

• Claude-SearchBot

  Documented job

  navigates the web to improve search result quality

  If you block it

  your pages leave Claude's search index

• Claude-User

  Documented job

  fetches a page when a person asks Claude a question

  If you block it

  Claude cannot read your page for that person

Each is addressed by name in robots.txt. Anthropic's documentation gives Crawl-delay as the way to slow ClaudeBot down and commits to respecting it where appropriate; Crawl-delay is a non-standard extension, so that is a stated commitment rather than a protocol rule. The crawler directory has a page for each agent: ClaudeBot, Claude-SearchBot, Claude-User.

How do you verify Anthropic's crawlers?

By IP, not by name. A user agent string is a header anyone can send. Anthropic publishes the IP addresses its crawlers use at claude.com/crawling/bots.json, so a log entry claiming to be ClaudeBot can be checked against that list. A firewall rule that trusts the header alone will let impostors in and may still block the real agent if it is written too broadly.

What are the common mistakes with Anthropic's crawlers?

One rule for all of Anthropic. A broad block meant to stop training also stops Claude-SearchBot and Claude-User, which removes you from answers as a side effect.

Assuming training equals visibility. Training shapes what Claude knows from memory, months later. Search and user fetches shape answers now. See AI training data.

Reading low crawl volume as low interest. Our log shows 104 Anthropic requests in 30 days, and about 35% of them were people asking. A small number of user-triggered fetches can matter more than thousands of training crawls.

Treating "Claude said" as one thing. Claude is a family of models, with and without web search, and answers vary from run to run. See context window for why even the largest models still select what they read.

How does Attensira track Claude?

Claude is one of the four engines a new Attensira workspace reads, through Anthropic's API: Attensira asks your buyers' questions, stores Claude's answers and records whether you are named and linked, with the sample size. Our own workspace reads only ChatGPT and Perplexity, so this page shows no Claude answer rate; not read is not zero. Attensira's crawler logs show which Anthropic agents fetched which of your pages, by the user agent each request claimed. The crew then fixes what blocks you and ships each change after your approval. Claude visibility covers what Attensira reads on Claude and how a change there is measured.