# PerplexityBot vs Perplexity-User: the crawler and the fetcher

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> PerplexityBot builds Perplexity's index and obeys robots.txt; Perplexity-User fetches for a person and generally does not. What that means for your site.

- Canonical page: https://attensira.com/compare/perplexitybot-vs-perplexity-user
- Family: compare
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=perplexitybot-vs-perplexity-user

**In short:** PerplexityBot crawls the web to build the index Perplexity searches, and it obeys robots.txt. Perplexity-User fetches a page when a person's question needs it, and Perplexity says it generally ignores robots.txt because a user asked. Allow PerplexityBot to be found. Expect Perplexity-User regardless, and use a firewall rule if you must stop it.

**Readings, 3 Sep 2026 to 4 Oct 2026**

| Measure | Value |
| --- | --- |
| of top-10k domains disallow PerplexityBot at the root (655) (n=8,036 domains) | 8.2% |
| disallow Perplexity-User, which generally ignores the rule (461) (n=8,036 domains) | 5.7% |
| requests labelled Perplexity-User on attensira.com, 30 days (n=143,601 AI-crawler requests) | 0 |

_Attensira robots.txt study and attensira.com AI traffic log · 4 Oct 2026 · n=8,036 domains; n=143,601 requests_

## Verdict

Same operator, two agents, two different relationships with robots.txt. PerplexityBot is the crawler your file governs and the one that gets you into Perplexity's results. Perplexity-User is a person's request, and Perplexity documents it as generally ignoring robots.txt.

## The readings on this page

### The readings at the top

Attensira robots.txt study of the Tranco top-10,000 (3 Sep 2026) and the attensira.com AI traffic log read over the Attensira MCP (5 Sep to 4 Oct 2026) · 3 Sep 2026 to 4 Oct 2026

| Reading | Value | Sample |
| --- | --- | --- |
| of top-10k domains disallow PerplexityBot at the root (655) | 8.2% | n=8,036 domains |
| disallow Perplexity-User, which generally ignores the rule (461) | 5.7% | n=8,036 domains |
| requests labelled Perplexity-User on attensira.com, 30 days | 0 | n=143,601 AI-crawler requests |

### How many large sites say anything to Perplexity-User?

robots.txt of the top-10,000 Tranco domains (list K9QPW), read by Attensira · 3 Sep 2026 to 3 Sep 2026

| Reading | Value | Sample |
| --- | --- | --- |
| never mention Perplexity-User (7,454) | 92.8% | n=8,036 domains |
| allow Perplexity-User by name (121) | 1.5% | n=8,036 domains |

## How do PerplexityBot and Perplexity-User compare?

Each cell cites a claim in the Sources section below.

| Criterion | PerplexityBot | Perplexity-User |
| --- | --- | --- |
| **What it is** |  |  |
| What triggers the fetch | Perplexity's own indexing schedule | A person asking Perplexity a question |
| Published user-agent token | ...compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot | ...compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user |
| Published address range list | perplexity.com/perplexitybot.json | perplexity.com/perplexity-user.json |
| Documented as a training crawler | No · Not used for foundation models, per Perplexity | No · Documented as answering a user's question |
| **What your controls do** |  |  |
| Honours robots.txt | Yes | No · Generally ignores it, per Perplexity |
| Produces a link back to your page | Yes · Surfaces and links sites in search results | Yes · May include a link in the response |
| What Perplexity recommends | Allow it, and its IP ranges, to appear in results | Nothing; it follows the user's request |
| How long a change takes | Up to 24 hours | Up to 24 hours |
| **What we measured** |  |  |
| Top-10k domains disallowing it at the root (study, n=8,036) | 655 (8.2%) | 461 (5.7%) |
| Requests to attensira.com, 30 days to 4 Oct 2026 | 621 requests, 331 pages | 0 labelled (n=143,601 AI-crawler requests) |

## What is PerplexityBot?

Perplexity's search crawler. It exists to surface and link websites in Perplexity search results, is documented as obeying robots.txt, and is explicitly not used to crawl content for foundation models.

## What is Perplexity-User?

Perplexity's user-initiated fetcher. It visits a page in the moment a person asks a question, and Perplexity documents it as generally ignoring robots.txt because a user requested the fetch.

## When should you choose PerplexityBot?

Write your robots.txt for PerplexityBot. It decides whether Perplexity's index holds your pages, it obeys your file, and Perplexity says it is not used to train foundation models, so allowing it costs you nothing on that front.

## When should you choose Perplexity-User?

Plan for Perplexity-User rather than against it. It arrives when a person's question needs your page, may link to that page in the answer, and generally ignores robots.txt, so a Disallow line under its name is a statement of preference, not a control.

## When is neither the right choice?

If a page must be kept away from both agents, robots.txt is not enough: put it behind authentication or a firewall rule matched on the user agent and Perplexity's published address lists. Neither agent decides whether Perplexity chooses to cite you; the page does.

## Where does Attensira fit?

Worth a look if you want both agents counted in your own logs (Growth and Business) next to a daily reading of what Perplexity cites. Whatever it takes to win the answer.

## What sets PerplexityBot and Perplexity-User apart?

- This is the one pair in the family where the operator documents its own agent as generally ignoring robots.txt, so the fix for an unwanted fetch lives in the firewall, not in the file.
- Perplexity's own answer to "list of AI crawler user agents" left Perplexity-User out on 4 October 2026, so a list copied from an answer can miss the one agent a Disallow does not stop.
- Neither Perplexity agent is documented as a training crawler, so the usual reason to block an AI crawler does not apply to either side of this pair.

## Why does Perplexity run two agents?

Because two different things happen when Perplexity answers a question. Before anyone asks, Perplexity needs an index of the web to search. PerplexityBot builds that index: Perplexity describes it as designed to surface and link websites in search results on Perplexity. Then, when a person asks a question that needs a specific page, Perplexity may read that page on the spot. Perplexity-User does that, and Perplexity says it may include a link to the page in its response.

The split mirrors what every answer engine does, but Perplexity documents it more plainly than most, including the uncomfortable part. Because a user requested the fetch, Perplexity says Perplexity-User generally ignores robots.txt rules. PerplexityBot, the scheduled crawler, follows them.

## What does Perplexity's own answer leave out?

We track the prompt "list of AI crawler user agents" every day. On 4 October 2026 Perplexity answered it with a list that named its own crawler, PerplexityBot, and the user fetchers of two other operators, ChatGPT-User and Claude-User, but not Perplexity-User. The engine described the industry's fetchers and skipped its own.

That is not a scandal, it is a retrieval artefact. Perplexity built the answer from third-party compilations, not from its own documentation. Across 30 Perplexity answers to that prompt in the 30 days to 4 October 2026, searchenginejournal.com took 28 of 298 citations (9.4%) and technologychecker.io took 24. Perplexity's own crawler page was not among the six most-cited sources. If a list your team copies from was written the same way, it may be missing the agent that matters most for a robots.txt decision: the one that ignores the file.

The practical rule is to write policy from the operator's page, not from a roundup. Perplexity's page names both agents, both user-agent strings and both address lists. A roundup names whatever its author found, on the day they found it. None of those 30 answers named or cited Attensira, which is the normal state for a prompt that does not ask about us by name.

## What does PerplexityBot control?

Whether your pages are in the index Perplexity searches. Perplexity's recommendation is direct: to appear in search results, allow PerplexityBot in robots.txt and permit requests from its published IP ranges. The second half matters. A bot-management rule that challenges unknown automated traffic can block PerplexityBot while your robots.txt invites it, and a challenge page is something a crawler cannot pass.

PerplexityBot is also not a training crawler. Perplexity states that it is not used to crawl content for AI foundation models. That removes the most common reason sites give for blocking AI crawlers: if your objection is to training, PerplexityBot is not the agent it is about.

## Why can Perplexity-User ignore robots.txt?

Because it acts for a person. When someone asks Perplexity a question and the answer needs your page, it fetches that page. The argument is the one browsers rely on: robots.txt governs automated crawling, and a fetch a person asked for is closer to a visit than to a crawl.

You do not have to accept that argument to plan around it. A Disallow group naming Perplexity-User records your preference, and some well-run sites write one for that reason. It is not a control. If a page must not be fetched by Perplexity-User, put it behind authentication or block it at the firewall, matched on the user-agent token and on the address list Perplexity publishes for this agent.

## How many large sites block each one?

On 3 September 2026 we read the robots.txt of the top 10,000 domains in a dated Tranco list and could read 8,036 of them. 655 (8.2%) disallow PerplexityBot at the site root. 461 (5.7%) disallow Perplexity-User, a rule Perplexity says the agent generally does not follow, and 92.8% never mention Perplexity-User at all.

The second number is the useful one. Hundreds of large sites have written a line that expresses a preference and may be relied on internally as a block. We do not know how many of them also enforce it at the firewall, and we will not guess. If yours is one of them, the check is simple: look for Perplexity-User requests from Perplexity's published addresses after the date the rule went in.

### How many large sites say anything to Perplexity-User?

Almost none. Of 8,036 top-10,000 domains whose robots.txt we read on 3 September 2026, nine in ten never name Perplexity-User, and only 121 allow it by name. Its behaviour is decided by Perplexity's policy, not by the file.

**Readings, 3 Sep 2026**

| Measure | Value |
| --- | --- |
| never mention Perplexity-User (7,454) (n=8,036 domains) | 92.8% |
| allow Perplexity-User by name (121) (n=8,036 domains) | 1.5% |

_Attensira robots.txt study, run of 3 September 2026 · 3 Sep 2026 · n=8,036_

## What did each agent fetch on attensira.com?

Attensira counts crawler requests at the edge on attensira.com, the way its AI traffic report does for customers. In the 30 days to 4 October 2026, PerplexityBot made 621 requests across 331 distinct pages, out of 143,601 AI-crawler requests in total. No request was labelled Perplexity-User. Perplexity's whole footprint on our site was search indexing.

Three things follow. Access was not our constraint: PerplexityBot read 331 of our pages, including this comparison's neighbours. Live fetches did not happen, which fits a small site that Perplexity answers from its index rather than one people ask it to open. And the citation rate is a content question, not an access one: over the same window Perplexity cited an attensira.com page in 85 of 2,175 answers (3.9%) to the 61 prompts we track.

One caution on timing. 616 of the 621 PerplexityBot requests fell in the final seven days of the window, so the monthly total is a burst, not a steady rate. User agents are self-reported and none of these requests was IP-verified, so read the split as a direction, not an audit.

## What should your robots.txt and firewall say?

For a site that wants to be cited by Perplexity, the robots.txt answer is short:

```
User-agent: PerplexityBot
Allow: /
```

A Perplexity-User group is optional. If you write one, add a comment that the real enforcement, if any, lives in your firewall:

```
# Preference only: Perplexity-User generally ignores robots.txt.
# Enforced at the CDN for /members/ (rule: block-pplx-user).
User-agent: Perplexity-User
Disallow: /members/
```

Check the wildcard group as well. A crawler obeys the group that names its token and falls back to `User-agent: *` only when no group names it. A years-old wildcard that disallows the root therefore blocks PerplexityBot unless PerplexityBot has its own group, and it is the most common way a site removes itself from Perplexity without anyone deciding to. robots.txt is also per host: a blog on a subdomain needs its own file carrying the same decision.

If you run bot protection at a CDN, add PerplexityBot's published addresses to its allow list and decide deliberately what to do with Perplexity-User's. The worst outcome is a file that seems to block something it does not, read by a colleague who then believes the problem is handled.

## How do you verify each one?

By user agent first, then by address. Perplexity publishes a separate address list for each agent:

| Check | PerplexityBot | Perplexity-User |
| --- | --- | --- |
| Token in the user agent | PerplexityBot/1.0 | Perplexity-User/1.0 |
| URL in the user agent | perplexity.ai/perplexitybot | perplexity.ai/perplexity-user |
| Address list to match | perplexitybot.json | perplexity-user.json |
| Obeys robots.txt | Yes | Generally not |

A request that claims either name from an address on neither list is not from Perplexity, and is the one you can block without a second thought.

Refresh the lists on a schedule. If you allowlist PerplexityBot by address at the firewall, a stale copy blocks the real crawler the day its ranges change, while your robots.txt still says allow. Count both agents at the server or edge: neither runs JavaScript, so neither appears in browser analytics.

## How should you report Perplexity in analytics?

As two lines, never one. A PerplexityBot request says Perplexity is re-reading part of your site; it is a coverage signal, useful for spotting sections the crawler never reaches, and nothing more. A Perplexity-User request says a person, at that moment, was reading an answer that needed your page. The second is rarer and closer to a visit, and adding it to the first hides the only part of the traffic that tracks real demand.

Neither line tells you whether you were cited. Crawling and citing are separate pipelines on separate schedules: a page can be fetched constantly and never cited, or cited from an index copy that is weeks old. Read requests as evidence of access, and read answers, sampled daily for the questions your buyers ask, as evidence of visibility. On our own site the two disagree in exactly the way you would expect: 331 pages read by the crawler, a small share of answers citing any of them.

## Is a fetcher that ignores robots.txt a problem?

It depends on what you used robots.txt for:

| If robots.txt was there to... | Perplexity-User is... | What to do |
| --- | --- | --- |
| Manage crawl load | A small concern: one page, because one person asked | Nothing |
| Keep a section out of search | Irrelevant: it does not build the index | Write the rule for PerplexityBot |
| Keep content out of AI products altogether | A real gap robots.txt cannot close | Authentication or a firewall rule |
| Protect paywalled or licensed text | A gap, like any fetch on a person's behalf | Put it behind authentication |

There is also a case for leaving it alone. A live fetch is the moment an answer engine reads your current page instead of an older copy. If your pricing changed last week, a user fetch is how the answer picks up the new price.

For most commercial pages the decision is: allow PerplexityBot and its addresses, leave Perplexity-User alone unless it causes a real problem, and spend the effort where citations come from, which is pages that answer your buyers' questions better than the pages Perplexity cites today.

## FAQ

### Does Perplexity-User obey robots.txt?

Generally not. Perplexity's documentation says that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules. A Disallow group naming it records your preference, but it is not a reliable block. To stop it, use a firewall rule matched on its user agent and on the address list Perplexity publishes for it.

### Does blocking PerplexityBot remove my site from Perplexity?

It removes the crawler that builds Perplexity's index, and Perplexity recommends allowing PerplexityBot and its published IP ranges to ensure a site appears in its search results. A site that blocks it should not expect to be surfaced, although Perplexity-User may still fetch a page a person asks about.

### Is either Perplexity agent used to train AI models?

Perplexity says PerplexityBot is not used to crawl content for AI foundation models, and documents Perplexity-User as fetching pages to answer a user's question. Neither is described as a training crawler, so there is no Perplexity training token to block.

### How do I tell PerplexityBot and Perplexity-User apart in my logs?

By the token in the user agent: PerplexityBot/1.0 or Perplexity-User/1.0, each with its own perplexity.ai URL. Then check the source address against the matching list Perplexity publishes, perplexitybot.json or perplexity-user.json. A request with the right name from the wrong address is not from Perplexity.

### How often does each Perplexity agent visit a small site?

On attensira.com, in the 30 days to 4 October 2026, PerplexityBot made 621 requests across 331 pages and no request was labelled Perplexity-User, out of 143,601 AI-crawler requests in total. Crawler visits follow Perplexity's schedule; user fetches follow people, so a quiet month for the fetcher is normal for a small site.

### How long do Perplexity crawler changes take to apply?

Perplexity says each setting works independently and that it may take up to 24 hours for its systems to reflect a change. Check your logs a day or two after any robots.txt or firewall edit, not the same afternoon.

## Sources

Each entry below is one claim on this page. `retrieved` is the date the exporter or a human last saw the source say this, and nothing more: it is not a publication date and not a promise of re-verification. `cite` is the anchor on the canonical page where this claim is rendered.

### pbot-purpose

Perplexity documents PerplexityBot as designed to surface and link websites in search results on Perplexity, and states it is not used to crawl content for AI foundation models.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pbot-purpose
- side: perplexitybot

> designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models
### pbot-robots

Perplexity documents PerplexityBot as obeying robots.txt.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pbot-robots
- side: perplexitybot
### pbot-ua

Perplexity publishes PerplexityBot's user-agent string as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pbot-ua
- side: perplexitybot
### pbot-ips

Perplexity publishes PerplexityBot's address ranges at perplexity.com/perplexitybot.json

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pbot-ips
- side: perplexitybot
### pbot-training

Perplexity states that PerplexityBot is not used to crawl content for AI foundation models.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pbot-training
- side: perplexitybot
### puser-purpose

Perplexity documents Perplexity-User as supporting user actions, visiting a web page to help answer a question and including a link to the page in its response.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-puser-purpose
- side: perplexity-user

> When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response
### puser-robots

Perplexity states that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-puser-robots
- side: perplexity-user

> Since a user requested the fetch, this fetcher generally ignores robots.txt rules
### puser-ua

Perplexity publishes Perplexity-User's user-agent string as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-puser-ua
- side: perplexity-user
### puser-ips

Perplexity publishes Perplexity-User's address ranges at perplexity.com/perplexity-user.json

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-puser-ips
- side: perplexity-user
### puser-links

Perplexity states that Perplexity-User may include a link to the page it visited in the response it produces.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-puser-links
- side: perplexity-user
### pbot-recommend

Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges to ensure a site appears in its search results.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pbot-recommend
- side: perplexitybot

> To ensure your site appears in search results, we recommend allowing PerplexityBot in your site's robots.txt file and permitting requests from our published IP ranges.
### pplx-propagation

Perplexity states that each setting works independently and that it may take up to 24 hours for its systems to reflect changes.

- source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (docs.perplexity.ai)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-pplx-propagation
- side: context
### study-pbot-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 655 (8.2%) disallow PerplexityBot at the site root.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-study-pbot-blocked
- side: perplexitybot
### study-puser-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 461 (5.7%) disallow Perplexity-User at the site root and 7,454 (92.8%) never mention it.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-study-puser-blocked
- side: perplexity-user
### dog-pbot

In the attensira.com workspace, read over the Attensira MCP for the 30 days to 4 October 2026, the AI traffic log recorded 621 PerplexityBot requests across 331 pages. User agents are self-reported and none was IP-verified.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-dog-pbot
- side: perplexitybot
### dog-puser

In the attensira.com workspace, read over the Attensira MCP for the 30 days to 4 October 2026, no request among 143,601 AI-crawler requests was labelled Perplexity-User.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/perplexitybot-vs-perplexity-user#claim-dog-puser
- side: perplexity-user