DuckAssistBot vs PerplexityBot: two crawlers built for cited answers

The short answer

These are the two crawlers on the open web with the least ambiguous case for allowing them. DuckAssistBot crawls in real time for DuckDuckGo's AI-assisted answers, which DuckDuckGo says prominently cite their sources. PerplexityBot exists to surface and link websites in Perplexity's results. Both operators have built products whose output points at the publisher, and both publish address lists so you can prove the traffic is real.

If you are triaging AI crawler decisions by risk, these two belong at the bottom of the list. The interesting question is not whether to allow them. It is what each operator has been willing to put in writing, because the two documents differ in ways that tell you something about each company.

What DuckDuckGo committed to on paper

DuckDuckGo's crawler page does three things almost nobody else does.

It states that the data is not used in any way to train AI models. That is an explicit negative, not an omission a publisher has to interpret. Most search crawlers operated by companies that also build models leave the boundary implicit, and a cautious publisher fills the silence with suspicion.

It states that opting out does not affect organic search rankings or result inclusion. DuckDuckGo runs a conventional search index alongside an AI answer layer, so it faced a question its pure-answer-engine peers do not: can a publisher refuse the AI feature without being punished in ordinary search? The answer is written down. Publishers who want to be in search results but not in AI summaries have a documented path.

And it states a number: robots.txt changes take effect after 72 hours. That single figure resolves the most common false alarm in crawler management, which is a site owner shipping a disallow, seeing the crawler again the next morning, and concluding the operator ignores the standard. Three days is the published window. Wait it out before escalating, and use the address list to confirm that what you are seeing is genuinely DuckAssistBot rather than something wearing its name.

What Perplexity committed to on paper

Perplexity's documentation is shorter and takes a different posture. It describes PerplexityBot as designed to surface and link websites in search results on Perplexity, and it recommends allowing PerplexityBot in your robots.txt.

That recommendation is worth pausing on. Every other operator in this space writes in the register of permission — here is the token, here is how to block it if you wish. Perplexity writes in the register of advocacy: allow this, it is in your interest. Whether you find that refreshing or presumptuous, it is a coherent position for a product whose entire output is a set of citations. Perplexity has less to gain from crawling a site that does not want to be quoted than a training-oriented operator does.

The other half of Perplexity's page is the part publishers should read carefully. Perplexity documents a second agent, Perplexity-User, and says it generally ignores robots.txt rules because the fetch originates from a user request. That disclosure is honest and it is also consequential: a robots.txt block on PerplexityBot stops the systematic crawl, and does not stop a Perplexity user's fetch of a specific page. DuckDuckGo's crawler page documents no equivalent agent, so a DuckAssistBot block has no documented bypass.

Where the two documents leave gaps

Neither page is complete, and it is worth naming what is missing rather than pretending the sourcing is exhaustive.

Perplexity's crawler documentation is scoped to search surfacing and does not make a statement about training the way DuckDuckGo's does. That is an absence, not a denial, and it should be read as exactly that: the page does not say the data trains models, and it does not say it does not. A publisher who needs that assurance in writing has it from one of these two operators and not the other.

DuckDuckGo's page does not state a propagation window for anything other than robots changes, and neither operator documents crawl-delay support. So for both agents, the vocabulary is allow or disallow, and any finer control is something you build at your own edge with rate limits.

Both publish addresses, which is the single most useful thing a crawler operator can do, and it means every claim about compliance on this page is checkable by a publisher with access logs and half an hour.

The robots.txt lines

Allow both, which is what the evidence supports for almost any site that wants to be found:

User-agent: DuckAssistBot
Allow: /

User-agent: PerplexityBot
Allow: /

Refuse DuckDuckGo's AI answers while staying in its conventional search index, which is the configuration DuckDuckGo's own documentation makes safe:

User-agent: DuckAssistBot
Disallow: /

Exclude a specific path from both — gated resources, customer lists, pricing experiments — while leaving your documentation and marketing pages available:

User-agent: DuckAssistBot
Disallow: /internal/

User-agent: PerplexityBot
Disallow: /internal/

Two mechanical notes. Robots.txt matches on the token, so the User-agent: line takes PerplexityBot and never the full Mozilla string. And rules are per host and per scheme: your documentation subdomain serves its own robots file and carries its own answer, and for most technical companies that host holds the pages an answer engine would most want to quote.

Version numbers, again

DuckAssistBot's published user agent carries /1.2; PerplexityBot's carries /1.0. Both numbers will change eventually, and any filter, firewall rule or dashboard written against the full string will stop matching on the day they do — silently, and in a way that looks exactly like a crawler losing interest in your site.

Match on the bare token and on the self-identifying URL, both of which are stable across versions. This is the single most common reason a company concludes that an AI crawler abandoned them, and it is a monitoring bug rather than a crawl event every time.

Two products, two very different reasons to be crawled

The crawlers look similar in a robots file. The products behind them are not, and the difference changes what allowing each one is worth to you.

DuckDuckGo is a search engine that added an answer layer. A user who asks it a question may get an AI-assisted answer with citations, or may get a conventional list of results, and DuckDuckGo has committed in writing that your presence in the second is unaffected by your decision about the first. The upside of allowing DuckAssistBot is therefore incremental: you were already in the index, and this adds a chance of appearing in the summarised answer above it.

Perplexity has no conventional results list to fall back on. The product is the answer, and the citations are the interface. A publisher who is not in Perplexity's crawl is not ranked lower — they are simply not among the sources the answer is built from. That makes the upside of allowing PerplexityBot categorical rather than incremental, and it explains why Perplexity's documentation asks rather than merely permits.

The practical consequence for prioritisation: if you have to argue one of these internally, argue Perplexity first, because the downside of being absent is total rather than partial. But the argument should rarely be necessary. Neither crawler asks for anything a publisher who wants readers should be reluctant to give.

The 72-hour rule is worth internalising

Most crawler escalations are premature. A team ships a robots.txt change, watches the logs for a day, sees the crawler again and starts drafting a complaint about an operator that ignores the standard.

DuckDuckGo has removed the ambiguity for its own crawler by publishing the window: 72 hours. Nothing you observe inside that period is evidence of anything. It is also a reasonable mental default for operators who publish no number at all, because the underlying mechanics are the same everywhere — robots files are fetched on a schedule and cached, and a crawl already scheduled against an older copy will still run.

The corollary is about the other direction, and it is the one people forget. An allow takes time to have an effect too. Opening access to a crawler that has been blocked for a year does not produce citations that week: the crawler has to return, fetch, and be selected as a source for a question somebody asks. Teams that flip the switch and check their numbers on Friday conclude the change did nothing. Give it a quarter, and measure whether the pages were fetched at all before drawing any conclusion about whether they were quoted.

What allowing them does not buy you

Being crawled is necessary for being cited and nowhere near sufficient. Both of these agents will happily fetch a page that neither product ever quotes, because the decision about what to quote is made downstream against whatever else is available on the question.

The work that closes that gap is content work, not robots work: does the page answer a question somebody actually asks, in a form that can be lifted into an answer without a human editor rewriting it, better than the pages currently being quoted. No directive in a robots file shortens that path, and any tool that implies otherwise is selling you a robots.txt editor with a dashboard attached.

It is also worth being precise about limits. Neither directive controls copies of your content that somebody else hosts under their own robots file. Neither governs what a person pastes into a chat window — Perplexity documents a separate agent for exactly that traffic. And neither is a licence: a robots directive is a machine-readable preference made to a well-behaved client, which both of these operators demonstrably are.

Confirming access, per URL

A robots.txt edit is an intention; your access log is the outcome. Both operators publish address lists, so for these two agents you can do the full verification: match the user-agent token, check the requesting address against the published file, and confirm which URLs were actually fetched.

That last part is where most of the value is. Knowing an answer-engine crawler reached your homepage tells you very little. Knowing whether it has read the twelve documentation pages that answer your buyers' real questions tells you whether you are a candidate source at all. Attensira's crawler logs record which AI agent fetched which URL and when. To read your current rules before changing them, the robots.txt generator and the bot access score will tell you what your file permits today.

Related comparisons

For PerplexityBot against the crawler behind ChatGPT's search features, read OAI-SearchBot vs PerplexityBot. For Perplexity's internal split between the crawler that honours robots.txt and the fetcher that does not, see PerplexityBot vs Perplexity-User. And for the two largest assistant vendors' search crawlers head to head, see Claude-SearchBot vs OAI-SearchBot.