PerplexityBot vs Perplexity-User: the crawler and the fetcher

Why does Perplexity run two agents?

Because two different things happen when Perplexity answers a question. Before anyone asks, Perplexity needs an index of the web to search. PerplexityBot builds that index: Perplexity describes it as designed to surface and link websites in search results on Perplexity. Then, when a person asks a question that needs a specific page, Perplexity may read that page on the spot. Perplexity-User does that, and Perplexity says it may include a link to the page in its response.

The split mirrors what every answer engine does, but Perplexity documents it more plainly than most, including the uncomfortable part. Because a user requested the fetch, Perplexity says Perplexity-User generally ignores robots.txt rules. PerplexityBot, the scheduled crawler, follows them.

What does Perplexity's own answer leave out?

We track the prompt "list of AI crawler user agents" every day. On 4 October 2026 Perplexity answered it with a list that named its own crawler, PerplexityBot, and the user fetchers of two other operators, ChatGPT-User and Claude-User, but not Perplexity-User. The engine described the industry's fetchers and skipped its own.

That is not a scandal, it is a retrieval artefact. Perplexity built the answer from third-party compilations, not from its own documentation. Across 30 Perplexity answers to that prompt in the 30 days to 4 October 2026, searchenginejournal.com took 28 of 298 citations (9.4%) and technologychecker.io took 24. Perplexity's own crawler page was not among the six most-cited sources. If a list your team copies from was written the same way, it may be missing the agent that matters most for a robots.txt decision: the one that ignores the file.

The practical rule is to write policy from the operator's page, not from a roundup. Perplexity's page names both agents, both user-agent strings and both address lists. A roundup names whatever its author found, on the day they found it. None of those 30 answers named or cited Attensira, which is the normal state for a prompt that does not ask about us by name.

How many large sites block each one?

On 3 September 2026 we read the robots.txt of the top 10,000 domains in a dated Tranco list and could read 8,036 of them. 655 (8.2%) disallow PerplexityBot at the site root. 461 (5.7%) disallow Perplexity-User, a rule Perplexity says the agent generally does not follow, and 92.8% never mention Perplexity-User at all.

The second number is the useful one. Hundreds of large sites have written a line that expresses a preference and may be relied on internally as a block. We do not know how many of them also enforce it at the firewall, and we will not guess. If yours is one of them, the check is simple: look for Perplexity-User requests from Perplexity's published addresses after the date the rule went in.

What did each agent fetch on attensira.com?

Attensira counts crawler requests at the edge on attensira.com, the way its AI traffic report does for customers. In the 30 days to 4 October 2026, PerplexityBot made 621 requests across 331 distinct pages, out of 143,601 AI-crawler requests in total. No request was labelled Perplexity-User. Perplexity's whole footprint on our site was search indexing.

Three things follow. Access was not our constraint: PerplexityBot read 331 of our pages, including this comparison's neighbours. Live fetches did not happen, which fits a small site that Perplexity answers from its index rather than one people ask it to open. And the citation rate is a content question, not an access one: over the same window Perplexity cited an attensira.com page in 85 of 2,175 answers (3.9%) to the 61 prompts we track.

One caution on timing. 616 of the 621 PerplexityBot requests fell in the final seven days of the window, so the monthly total is a burst, not a steady rate. User agents are self-reported and none of these requests was IP-verified, so read the split as a direction, not an audit.

What should your robots.txt and firewall say?

For a site that wants to be cited by Perplexity, the robots.txt answer is short:

User-agent: PerplexityBot
Allow: /

A Perplexity-User group is optional. If you write one, add a comment that the real enforcement, if any, lives in your firewall:

# Preference only: Perplexity-User generally ignores robots.txt.
# Enforced at the CDN for /members/ (rule: block-pplx-user).
User-agent: Perplexity-User
Disallow: /members/

Check the wildcard group as well. A crawler obeys the group that names its token and falls back to User-agent: * only when no group names it. A years-old wildcard that disallows the root therefore blocks PerplexityBot unless PerplexityBot has its own group, and it is the most common way a site removes itself from Perplexity without anyone deciding to. robots.txt is also per host: a blog on a subdomain needs its own file carrying the same decision.

If you run bot protection at a CDN, add PerplexityBot's published addresses to its allow list and decide deliberately what to do with Perplexity-User's. The worst outcome is a file that seems to block something it does not, read by a colleague who then believes the problem is handled.

How do you verify each one?

By user agent first, then by address. Perplexity publishes a separate address list for each agent:

• Token in the user agent

  PerplexityBot

  PerplexityBot/1.0

  Perplexity-User

  Perplexity-User/1.0

• URL in the user agent

  PerplexityBot

  perplexity.ai/perplexitybot

  Perplexity-User

  perplexity.ai/perplexity-user

• Address list to match

  PerplexityBot

  perplexitybot.json

  Perplexity-User

  perplexity-user.json

• Obeys robots.txt

  PerplexityBot

  Yes

  Perplexity-User

  Generally not

A request that claims either name from an address on neither list is not from Perplexity, and is the one you can block without a second thought.

Refresh the lists on a schedule. If you allowlist PerplexityBot by address at the firewall, a stale copy blocks the real crawler the day its ranges change, while your robots.txt still says allow. Count both agents at the server or edge: neither runs JavaScript, so neither appears in browser analytics.

How should you report Perplexity in analytics?

As two lines, never one. A PerplexityBot request says Perplexity is re-reading part of your site; it is a coverage signal, useful for spotting sections the crawler never reaches, and nothing more. A Perplexity-User request says a person, at that moment, was reading an answer that needed your page. The second is rarer and closer to a visit, and adding it to the first hides the only part of the traffic that tracks real demand.

Neither line tells you whether you were cited. Crawling and citing are separate pipelines on separate schedules: a page can be fetched constantly and never cited, or cited from an index copy that is weeks old. Read requests as evidence of access, and read answers, sampled daily for the questions your buyers ask, as evidence of visibility. On our own site the two disagree in exactly the way you would expect: 331 pages read by the crawler, a small share of answers citing any of them.

Is a fetcher that ignores robots.txt a problem?

It depends on what you used robots.txt for:

• Manage crawl load

  Perplexity-User is...

  A small concern: one page, because one person asked

  What to do

  Nothing

• Keep a section out of search

  Perplexity-User is...

  Irrelevant: it does not build the index

  What to do

  Write the rule for PerplexityBot

• Keep content out of AI products altogether

  Perplexity-User is...

  A real gap robots.txt cannot close

  What to do

  Authentication or a firewall rule

• Protect paywalled or licensed text

  Perplexity-User is...

  A gap, like any fetch on a person's behalf

  What to do

  Put it behind authentication

There is also a case for leaving it alone. A live fetch is the moment an answer engine reads your current page instead of an older copy. If your pricing changed last week, a user fetch is how the answer picks up the new price.

For most commercial pages the decision is: allow PerplexityBot and its addresses, leave Perplexity-User alone unless it causes a real problem, and spend the effort where citations come from, which is pages that answer your buyers' questions better than the pages Perplexity cites today.