AI crawler user agents and robots.txt tokens

What are the exact user-agent strings and robots.txt tokens for AI crawlers?

Each operator publishes two things per agent: a short product token that robots.txt groups are addressed to, and the full User-Agent header the agent actually sends. Write rules against the token. Match logs against the token as a substring. The tables below carry both, transcribed from each operator's own documentation on 3 September 2026.

How to read a crawler user-agent string

A modern crawler header has three parts, and only one of them identifies anything.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
└──────── browser compatibility preamble ────────┘  └── token+version ──┘  └──── docs URL ────┘

The preamble is inherited convention. Servers and frameworks have sniffed for Mozilla since the 1990s, and a crawler that omits it gets rejected by code nobody has looked at in a decade. It carries no information about the crawler.

The middle part is what matters: the product token, then a version. Match the token, never the version. OpenAI's agents carry versions such as GPTBot/1.4 and OAI-SearchBot/1.4,[^openai-searchbot-string] and those increment. A log filter or firewall rule pinned to a version stops matching silently the next time the operator ships.

The trailing +https:// URL is a convention worth using: it points at the operator's own documentation for that agent, which is the correct place to check an unfamiliar token rather than a third-party bot list.

OpenAI

OpenAI documents four agents, each with its own token, its own purpose and its own IP range file.[^openai-agents]

Full header strings, as documented — GPTBot's is the one diagrammed above:[^openai-gptbot-string]

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10157) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Two details are easy to miss. OAI-SearchBot sends a full desktop Chrome preamble while the others do not — so a firewall rule that blocks headless-looking agents can catch three of the four and miss the fourth, or the reverse. And OpenAI states that robots.txt rules may not apply to ChatGPT-User, because those fetches follow user actions, while the other three respect it.[^openai-chatgpt-user-robots]

The per-agent IP files matter more than they look. Because each agent has its own, you can verify not just that a request came from OpenAI but which of the four it was, independent of the header.[^openai-ip-files]

Anthropic

Anthropic documents three agents and one shared verification file.[^anthropic-tokens]

Two things distinguish this set. First, Claude-User is a user-triggered fetcher that Anthropic nonetheless documents as honouring robots.txt — the opposite of OpenAI's and Perplexity's stated position on their equivalents. Second, all three support the Crawl-delay extension, which most operators here do not mention.

Anthropic publishes its addresses at claude.com/crawling/bots.json, and adds a caveat worth heeding: it states that IP blocking may not work correctly and recommends robots.txt instead.[^anthropic-ip-blocking] The IP file is for verifying traffic, not for building a blocklist against.

Perplexity

Two agents, and the cleanest documented split in this chapter.[^perplexity-purposes]

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Perplexity documents no training crawler. PerplexityBot is stated as not used to crawl content for AI foundation models, which means a blanket AI block aimed at training removes only the agent capable of linking to you.[^perplexity-strings]

Google

Google's set is not organised the way the others are, because Search came first.[^google-tokens]

Google-Extended never appears in a log file. It changes what may be done with content that Googlebot already fetched. That is why disallowing it does not affect Search inclusion or ranking — there is no fetch to withhold.

Apple

Apple documents desktop and mobile header forms, both ending in the same identifying fragment:[^applebot-strings]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10157) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (iPhone; CPU iPhone OS 1741 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

Apple is the clearest illustration of the pattern this whole guide rests on: one crawler doing a search job whose output is separately licensed for a training job, with a distinct token for each decision.

The rest

Amazon's header carries a literal version placeholder as documented,[^amazonbot-string] which is another reason to match on the bare token. Common Crawl's is the shortest string in this chapter,[^ccbot-string] with no browser preamble at all. Meta's 24-hour caching window is worth knowing before you conclude a new rule is being ignored.[^meta-tokens] Mistral's negative statement — no automatic crawling, no training — is unusually specific, and the sort of thing to re-check at each quarterly review.[^mistral-string]

Tokens you may expect here and will not find

This chapter lists an agent only where its operator publishes documentation naming it and stating what it does. Several tokens that circulate widely on third-party bot lists are therefore absent, and the absence is deliberate rather than an oversight.

Some are search engine crawlers whose relationship to an AI product is real but not documented as a separate token, so a claim about what allowing them does for AI visibility would be inference dressed as fact. Others are commercial scrapers and content-extraction services whose own material describes a product rather than a crawling policy. A third group are tokens that appear in aggregated lists with no traceable operator page at all — the exact shape of a fact that has been copied between blog posts until it looks sourced.

The practical consequence for your robots.txt: an undocumented token can still be named in a rule, and naming it costs nothing. What you cannot do honestly is claim to know what the rule achieves. Where the operator has not said, this guide does not say either.

Separating the two kinds of traffic

The reason to get the tokens exactly right is that they are the only way to split your crawler traffic into the two groups that matter. Once matching is reliable, one grouping answers the question people actually have.

A site whose AI traffic is entirely in the second row has a configuration problem, not a content problem — and it is invisible unless the tokens are matched precisely enough to tell the rows apart.

Matching these in logs

Three rules keep a log filter working across a year of operator changes.

1. Match the bare token, case-insensitively. GPTBot, not GPTBot/1.4, and

not the whole header.

1. Treat the token as a whole word where you can. Claude-User is a

substring of nothing else here, but naive substring matching across a longer bot list eventually produces a collision.

1. Verify before you count. A header is free text and AI crawler tokens are

the ones scrapers most often borrow. A count of unverified GPTBot hits is a count of requests that claimed to be GPTBot. The verification chapter covers turning that claim into a fact.

Once you can match them reliably, the question becomes which of them you actually want. That is training crawlers versus search crawlers, and the directives that follow from it are in the robots.txt chapter.

[^openai-agents]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^openai-searchbot-string]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^openai-gptbot-string]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^openai-ip-files]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^openai-chatgpt-user-robots]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^anthropic-tokens]: Anthropic support documentation, retrieved 2026-09-03. [^anthropic-ip-blocking]: Anthropic support documentation, retrieved 2026-09-03. [^perplexity-purposes]: Perplexity crawler documentation, retrieved 2026-09-03. [^perplexity-strings]: Perplexity crawler documentation, retrieved 2026-09-03. [^google-tokens]: Google Search Central, common crawlers, retrieved 2026-09-03. [^applebot-strings]: Apple support, About Applebot, retrieved 2026-09-03. [^amazonbot-string]: Amazon developer documentation, Amazonbot, retrieved 2026-09-03. [^ccbot-string]: Common Crawl, CCBot, retrieved 2026-09-03. [^meta-tokens]: Meta for Developers, web crawlers, retrieved 2026-09-03. [^mistral-string]: Mistral documentation, robots, retrieved 2026-09-03.