PerplexityBot vs Perplexity-User: one obeys robots.txt, one does not

The short answer

Perplexity runs two agents with opposite robots.txt policies, and it says so plainly. PerplexityBot indexes pages so Perplexity can surface and link them in search results, and it obeys robots.txt. Perplexity-User visits a page because a person asked a question, and Perplexity documents it as generally ignoring robots.txt rules for exactly that reason.

Any sentence beginning "Perplexity respects robots.txt" or "Perplexity ignores robots.txt" is half right and half wrong. There is no single answer.

Why one operator takes both positions

The reasoning behind the split is not hidden. Perplexity's position is that a fetch requested by a person is different in kind from an automated crawl, and that robots.txt governs the second rather than the first. That is a defensible reading of what the standard was designed for — a mechanism for managing crawler load and crawl scope, written long before an assistant might fetch one page on one person's behalf.

It is also a contested reading, and worth naming as contested rather than settled. Anthropic reaches the opposite conclusion and documents its user-initiated agent as honouring robots.txt, describing the token as the mechanism by which site owners control user-initiated access. Two operators, the same situation, incompatible answers. A publisher does not have to pick a side in that argument, but they do have to know that it exists, because it determines whether their file does anything.

What PerplexityBot is and is not

PerplexityBot has a narrower remit than most agents with comparable name recognition. Perplexity states that it is designed to surface and link websites in search results on Perplexity, and states that it is not used to crawl content for AI foundation models. That second half is a real limitation on scope, published by the operator, and it removes the usual training objection from the conversation entirely.

What that leaves is an ordinary search-crawler decision. If Perplexity is a channel you want, the agent whose job is surfacing and linking you should be allowed to do it. Perplexity's documentation does not state an explicit consequence for blocking PerplexityBot the way OpenAI does for OAI-SearchBot, so this page will not claim one — but disallowing the crawler that exists to link to you is not a neutral act, and nobody should pretend otherwise.

What Perplexity-User is

Perplexity-User is a live fetch on a person's behalf. Somebody asked a question, Perplexity decided your page might answer it, and the agent went to look. Perplexity states that the response may include a link to the page it visited.

That is a very different economic transaction from a training crawl. A training crawler takes your content and gives nothing back. A user-initiated fetcher takes one page and may return an attributed link and a reader. For a company whose problem is being unknown, this is close to the best traffic on the internet: pre-qualified, intent-heavy, and arriving because somebody already wanted to know.

The counter-argument deserves fair statement. For publishers whose revenue is the visit itself, an answer that summarises the page reduces the number of people who arrive at it, and the attribution link does not fully replace what was lost. That grievance is real. It is also not the situation of most companies reading this, whose difficulty is the opposite one.

What you can and cannot do about it

Robots.txt directives, honest about what each achieves:

User-agent: PerplexityBot
Disallow: /

That one is expected to work — the operator documents the agent as obeying robots.txt.

User-agent: Perplexity-User
Disallow: /

That one is documented in advance as generally not working. Writing it is not harmful, and it records your preference for anyone reading the file, but it should not be mistaken for an outcome.

If you genuinely need Perplexity-User stopped, the instrument is enforcement rather than request: a rule at your server or edge that matches the agent and refuses it. Perplexity publishes address ranges for each agent separately, at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json, which is what makes such a rule possible to write accurately rather than by user-agent guesswork. It is worth being clear-eyed that this is a decision to turn away visitors a human being sent, and to make it deliberately rather than as a reflex.

The argument each side of this debate is actually making

It is worth laying out both positions properly, because the disagreement between operators is not carelessness on anybody's part and treating it as such leads to bad decisions.

The case for exempting user-initiated fetches runs like this. Robots.txt was written to manage automated crawling — the load, the scope, the parts of a site a machine should not walk. A person asking a question and an assistant fetching one page in response is not that. It is closer to a browser following a link, and nobody expects robots.txt to stop a browser. Extending an anti-crawling mechanism to cover a human's own request, on this view, would let a site block a person from reading a page they asked to read, which is not what the standard was for.

The case against is equally coherent. The fetch is not being made by a person; it is being made by a company's infrastructure, at that company's discretion, and the output is a summary that may substitute for the visit. Volume matters too: an assistant serving millions of users generates a pattern of traffic that no individual browser does, whatever the trigger. On this view, calling it user-initiated is an accounting choice about whose intent counts, and it happens to be the accounting that removes the publisher's veto.

Anthropic has taken the second view and applied it to its own live fetcher. OpenAI and Perplexity have taken the first. None of them is hiding what they do, which is the important part — the documentation is clear enough on all three sides that a publisher can find out where they stand rather than having to reverse-engineer it from logs.

What this means for how you write your file

Given all that, the useful discipline is to stop treating robots.txt as a single instrument with a single reliability, and start treating each token as a separate claim about the future.

Some of your rules are enforceable requests: PerplexityBot, GPTBot, ClaudeBot, Googlebot. The operator has said it will honour them, and if it does not, you have a specific, checkable grievance.

Some are declarations of preference: Perplexity-User and ChatGPT-User. The operator has told you in advance that the rule may not apply. Writing it is still worthwhile — it records intent, it costs nothing, and it is already in place if a policy changes — but you should not report it internally as a block, and you should not be surprised by fetches that follow.

And some are neither, because the token does not fetch anything at all. Google-Extended and Applebot-Extended sit in that category, governing use rather than access.

Three kinds of line in one file, with three different meanings and no visual distinction between them. That is the real usability problem with robots.txt in 2026, and until the standard catches up the workaround is a comment above each block saying which kind it is. A future colleague reading your file has no other way to tell.

Verifying what is actually happening

This pairing is the strongest argument in the whole family for looking at logs rather than files. When one agent honours your rules and the other announces that it generally will not, the robots file has stopped being a description of reality. The only description of reality is the record of what fetched what.

Both agents identify themselves clearly. Match on PerplexityBot/1.0 and Perplexity-User/1.0, or better on the self-identifying URLs, which do not change when a version increments. Then check the requesting addresses against the two published lists. Anything wearing a Perplexity user agent from outside those ranges is somebody else's scraper, and counting it as Perplexity traffic will give you a flattering and false picture of your visibility.

Attensira's crawler logs record these fetches per agent and per URL, which turns a policy question into an observation. The bot access score reads back what your current rules permit, and the robots.txt generator will produce a correct file if you decide to change one.

The decision, briefly

Allow PerplexityBot unless you have a specific reason to be absent from Perplexity's index. It does not train models, it exists to link to you, and your file genuinely controls it.

Treat Perplexity-User as traffic rather than as a crawler. It arrives because a person asked, it may cite you, and a robots directive will generally not stop it anyway. If your business model makes that intolerable, enforce at the edge using the published address list — and know that you are choosing to be absent from answers to questions people are already asking about you.

Related comparisons

The two user-initiated fetchers from the two largest assistants are compared directly in ChatGPT-User vs Perplexity-User, which is where the robots.txt disagreement is sharpest. For PerplexityBot against its closest equivalent at OpenAI, read OAI-SearchBot vs PerplexityBot. Anthropic's opposite policy on user-initiated fetching is covered in ClaudeBot vs Claude-User, and OpenAI's training-versus-search split in GPTBot vs OAI-SearchBot.