OAI-SearchBot vs PerplexityBot: ChatGPT's and Perplexity's search crawlers

Why compare two crawlers from different companies?

Because they are the two that decide whether your page can be a cited source in the two answer engines that show sources most consistently. OAI-SearchBot fetches pages for ChatGPT's search features. PerplexityBot fetches pages for Perplexity's search results. Each is the reason a link to your site can appear under an answer, and each is documented by its operator as separate from training foundation models.

Most crawler comparisons are about one company's agents: training against search, or crawler against live fetcher. This one compares like with like across vendors. The useful question is not which to allow. If you want to be cited, you allow both. The useful questions are how each vendor describes the cost of blocking, how you verify each one, and what each actually does on a real site.

What does each operator promise?

OpenAI describes OAI-SearchBot as the crawler used to surface websites in search results in ChatGPT's search features, and states that sites opted out of it will not be shown in ChatGPT search answers. Training sits under a different crawler, GPTBot, and OpenAI states that a site can allow OAI-SearchBot while disallowing GPTBot. The promise is structural: training and search are different tokens, so you can choose them separately.

Perplexity describes PerplexityBot as designed to surface and link websites in search results on Perplexity, and says it is not used to crawl content for AI foundation models. Its documentation adds a recommendation: to ensure your site appears in search results, allow PerplexityBot in robots.txt and permit requests from Perplexity's published IP ranges. The promise is stated rather than structural: one crawler, with a written boundary.

Both say changes take about a day. OpenAI gives about 24 hours for OAI-SearchBot; Perplexity says it may take up to 24 hours for its systems to reflect changes.

What does blocking each one cost?

Blocking OAI-SearchBot has a documented cost: your site stops appearing in ChatGPT search answers. Blocking PerplexityBot has a cost Perplexity describes indirectly: if allowing it is how you ensure you appear, blocking it is how you ensure you may not. Neither vendor publishes how quickly an existing citation disappears after a block, and answers lag behind crawl decisions.

The less obvious cost is the one nobody writes into robots.txt. Perplexity asks site owners to permit its IP ranges, which is a firewall instruction, not a robots.txt one. A bot-management rule at a CDN that challenges unknown automated traffic can block PerplexityBot or OAI-SearchBot while robots.txt says allow. The file is a request to well-behaved crawlers; the firewall is what actually answers it.

What do the robots.txt lines look like?

For a site that wants to be cited by both engines:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

If your wildcard group already allows the root, these groups are optional, but naming them protects you from a future wildcard edit that disallows more than intended. If a section should never be quoted, disallow it in both groups rather than one, so the two engines see the same site.

What did they actually fetch from us?

Attensira counts crawler requests at the edge on attensira.com, the same way it does for customers on Growth and Business. In the 30 days to 4 October 2026, 621 requests were labelled PerplexityBot, across 331 distinct pages. In the same window 7,536 requests carried OpenAI user agents, and about 86% of those, roughly 6,480, were classed as search indexing, which is OAI-SearchBot's job. On our site the OpenAI search crawler was the busier of the two by about ten to one.

The timing matters as much as the totals. 616 of PerplexityBot's 621 requests fell in the last 7 days of the window, so for most of the month it barely visited, then read our site in a burst. A team that sampled one quiet week would conclude Perplexity ignores them; a team that sampled the last week would overstate its monthly rate about fourfold. Read crawler volume over a month, not a week.

No request was labelled Perplexity-User in the window. That is the agent Perplexity uses when a person asks it to read a specific page, so its absence says something about how people reached us, not about PerplexityBot. Every label is the user agent the request claimed; none was IP-verified, and our documentation says we do not claim verified attribution.

What about the agents that fetch on a person's behalf?

Each company runs a second agent that is easy to confuse with its search crawler. OpenAI's ChatGPT-User fetches a page when a person in ChatGPT asks for it, and OpenAI says robots.txt rules may not apply to it. Perplexity's Perplexity-User visits a page when a person asks Perplexity a question that needs it, and generally ignores robots.txt because a user requested the fetch.

Those agents matter for a different reason. A live fetch is a person, right now, reading an answer that involves your page. But it is not something you can reliably grant or deny through robots.txt, and it is not what puts you in the index each engine searches. Keep the two kinds apart in your logs and your policy: crawlers you allow or block in robots.txt, live fetchers you handle, if at all, at the firewall.

How do large sites treat the two?

On 3 September 2026 we read the robots.txt files of the top 10,000 domains in Tranco list K9QPW and parsed 8,036 of them. 467 (5.8%) disallow OAI-SearchBot at the root and 655 (8.2%) disallow PerplexityBot. Explicit allows are rarer and almost level: 335 (4.2%) allow OAI-SearchBot by name and 313 (3.9%) PerplexityBot. The large majority name neither and fall back to their wildcard group.

Our data cannot say why the block rates differ, and we will not guess at motives. What it does say is that a PerplexityBot block is more common among large sites than an OAI-SearchBot block, so if you inherited a robots.txt with one, check whether the reason still applies. The full table is in our study of AI crawler access.