For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Bots

User-initiated fetchers

User-initiated bots fetch one page because a person just asked an assistant about it: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User and Amzn-User. This is the most direct path from your page to a cited answer. OpenAI, Perplexity and Amazon say robots.txt may not apply; Anthropic says Claude-User respects it, and Mistral says its token governs which sites these requests reach.

Start freeBook a demo

7 days free · card required

User-initiated fetchers5 agents
agents documented
5
publish IP ranges or reverse DNS
5 of 5
requests to attensira.com, 30 days
≈270
of n=143,601 AI crawler requests
of top sites disallow ChatGPT-User, the most blocked of the 4 in the study
7.1%
n=8,036 robots.txt files

One of the category's rules, from the ChatGPT-User entry, per the operator's documentation; the other 4 are on their own entries.

User-agent: ChatGPT-UserDisallow: /private/
Researcher agentResearcherOperator documentation, our AI traffic board (Attensira MCP) and robots.txt study ·

Which user-initiated fetchers are documented?

Fetches one page because a person just asked about it, so blocking it turns that reader away.

Which user-initiated fetchers fetched attensira.com in the last 30 days?

2 of 5 had requests filed on attensira.com in the window. User agents as sent, not checked against address lists.

See which crawlers visit your site.

AI traffic · attensira.com2 of 5 seen
Requests to attensira.com by agent, last 30 days to 4 Oct 2026, and attensira.com's robots.txt rule for each
AgentRequestsOur robots.txt
ChatGPT-User≈230Allowed by name
Claude-User≈36Allowed by name
MistralAI-User0Allowed by name
Perplexity-User0Allowed by name
Amzn-User0No named group
AI traffic board, read over the Attensira MCP · · n=143,601 requests

How do the user-initiated fetchers compare?

Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.

Sorted by top sites blocking, high to low.

  • ChatGPT-User

    Operator
    OpenAI
    robots.txt
    May not apply
    Verify by
    IP ranges
    attensira.com
    Allowed by name
    Top sites blocking
    7.1%

  • Perplexity-User

    Operator
    Perplexity
    robots.txt
    May not apply
    Verify by
    IP ranges
    attensira.com
    Allowed by name
    Top sites blocking
    5.7%

  • Claude-User

    Operator
    Anthropic
    robots.txt
    Respected
    Verify by
    IP ranges
    attensira.com
    Allowed by name
    Top sites blocking
    5.5%

  • MistralAI-User

    Operator
    Mistral AI
    robots.txt
    Respected
    Verify by
    IP ranges
    attensira.com
    Allowed by name
    Top sites blocking
    5.4%

  • Amzn-User

    Operator
    Amazon
    robots.txt
    May not apply
    Verify by
    IP ranges
    attensira.com
    No named group
    Top sites blocking
    Not in study

Each row read from the operator's documentation, 3 Sep 2026.

“Not in study” means the token is not one our robots.txt study counts yet, not that no site blocks it.

Why is a user-initiated fetch the closest thing to a citation?

Because it happens inside an answer. A person asks a question, the assistant decides your page might answer it, and the fetcher reads it right then. Mistral documents that the answer then includes a link to the source. A training crawl may never surface your page; this fetch exists to quote it.

User-AgentChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

The version number in this string changes; match the token, not the whole string. The ChatGPT-User entry, and reading user-agent strings.

Researcher agentResearcherTranscribed from OpenAI documentation ·

Should I block user-triggered fetchers?

Rarely, and robots.txt may not do it anyway. OpenAI states that robots.txt rules may not apply to ChatGPT-User, Perplexity documents that Perplexity-User generally ignores robots.txt, and Amazon says Amzn-User may not follow all directives. Anthropic says Claude-User honors robots.txt, and Mistral says MistralAI-User governs which sites these requests can reach.

Where a block must hold, it has to be a network rule. OpenAI, Perplexity, Amazon and Mistral publish a separate address file for their user fetcher, which is what turns a preference into a block. Anthropic's one file covers all three of its agents, so it proves the request is Anthropic's but not which agent sent it.

VerificationPerplexity-User
IP range file
perplexity-user.json

Perplexity publishes no reverse-DNS suffix for Perplexity-User; verify by the range file.

A per-agent address file is the only enforceable control here. Because the fetcher is documented as generally ignoring robots.txt, a network rule built on this file is the difference between a preference and a block. The Perplexity-User entry, and the method step by step: how to verify a crawler.

Researcher agentResearcherTranscribed from Perplexity documentation ·

How often do the most visited sites block user-initiated fetchers?

Disallowed at the rootn=8,036 robots.txt files
Top 10,000 domains (Tranco), by agent · run
NameDisallowed at the root
ChatGPT-User7.1%
Perplexity-User5.7%
Claude-User5.5%
MistralAI-User5.4%
Web engineer agentWeb engineer · Writes robots rules like theseRobots.txt access study, our own fetch · · n=8,036 robots.txt files

ChatGPT-User is the most blocked here: 573 of 8,036 sites disallow it at the root. The least blocked, MistralAI-User, is disallowed by 434.

A site that never names an agent is counted as unspecified, not as allowing it.

What else do people ask?

What is a user-initiated bot?
A fetcher that reads one page because a person just asked an AI assistant something that page might answer. ChatGPT-User, Claude-User and Perplexity-User are examples. It is not a crawl: one request, for one question.
Does ChatGPT-User obey robots.txt?
OpenAI states that robots.txt rules may not apply to ChatGPT-User, because the fetch follows a user action. A Disallow naming it is a request; the enforceable control is a network rule against OpenAI's chatgpt-user.json ranges.
How is Claude-User different from ClaudeBot?
ClaudeBot is Anthropic's training crawler. Claude-User fetches a page when a person asks Claude a question, and Anthropic says it honors robots.txt. You can disallow ClaudeBot and allow Claude-User in separate groups.
Does blocking user-triggered fetchers protect my content?
It mostly removes you from live answers. These fetches read one page for one question; blocking them does not keep you out of training, which has its own tokens, and for three of the five operators robots.txt may not stop them at all.

Where do these facts come from?

5 claims on this page, each with the page it was read from; last checked .

  1. Mistral states that MistralAI-User is for user actions in its assistant, and that when users ask a question it may visit a web page to help answer and include a link to the source in its response.

    docs.mistral.ai · retrieved

    “When users ask Vibe a question, it may visit a web page to help answer and include a link to the source in its response.”

  2. Perplexity documents Perplexity-User as supporting user actions within Perplexity, visiting a web page when users ask a question to help provide an accurate answer and including a link to the page in its response.

    docs.perplexity.ai · retrieved

    “Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.”

  3. Amazon states that Amzn-User supports user actions, such as responding to Alexa queries that require up-to-date information, and may fetch live information from the web to provide accurate answers on the user's behalf.

    developer.amazon.com · retrieved

    “Amzn-User supports user actions, such as responding to Alexa queries that require up-to-date information.”

  4. OpenAI documents ChatGPT-User as the agent used for certain user actions in ChatGPT and Custom GPTs, visiting a web page when a user asks a question.

    developers.openai.com · retrieved

    “used for certain user actions in ChatGPT and Custom GPTs”

  5. Anthropic states that its bots, Claude-User among them, respect "do not crawl" signals by honouring industry standard directives in robots.txt.

    support.claude.com · retrieved

    “Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt”

Want whatever it takes to win the answer?

Start freeBook a demo

7 days free · card required