AI crawler user agents and robots.txt tokens

Which crawler tokens actually show up in a real log?

The chart at the top of this page is the ones that reached our own site, by the token each request carried, in the 30 days to 4 October 2026. Attensira counts crawler requests to attensira.com at the edge, because AI crawlers do not run JavaScript and never trigger browser analytics. One operator's token carried about nine in every ten requests; Anthropic's two tokens carried fewer than a hundred between them.

These are counts of what each request said it was. A header is free text, so they are not proof of who sent it; the verification chapter covers that. They are, though, exactly what your robots.txt rules are matched against.

Then we checked our own robots.txt against the operators' current documentation, and found it had drifted: six tokens out of step, two of them stale and four missing. The band after the Anthropic and Mistral tables below lists them. Agents without a group of their own fall to our wildcard group, which allows most of the site. That happens to be what we want. It would not be, for a site that meant to block training.

The lesson is the reason this chapter exists: tokens change, operators add agents, and a robots.txt written once drifts out of date. The tables below come from each operator's own documentation, fetched on 4 October 2026.

How do you read an AI crawler's user-agent string?

A modern crawler header has three parts, and only one of them identifies anything.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
└──────── browser compatibility preamble ────────┘  └── token/version ──┘  └──── docs URL ────┘

The preamble (Mozilla/5.0 ...) is inherited convention. Servers have sniffed for "Mozilla" since the 1990s, and a crawler without it gets turned away by old code. It says nothing about the crawler.

The token and version (GPTBot/1.4) are what matter. robots.txt groups are addressed to the token. Match the token, never the version: versions change when the operator ships, and a firewall rule or log filter pinned to GPTBot/1.4 stops matching silently on the next release.

The trailing URL points at the operator's own page for that agent. When an unfamiliar token turns up in your logs, that page, not a third-party bot list, is the place to check what it does.

What are OpenAI's crawler tokens?

OpenAI documents four agents, each with its own token, job and IP list.

• OAI-SearchBot

  Job

  Surface websites in ChatGPT's search features

  Follows robots.txt

  Yes

  IP list

  openai.com/searchbot.json

• GPTBot

  Job

  Make its foundation models more useful and safe (training)

  Follows robots.txt

  Yes

  IP list

  openai.com/gptbot.json

• ChatGPT-User

  Job

  Certain user actions in ChatGPT and custom GPTs

  Follows robots.txt

  Rules may not apply

  IP list

  openai.com/chatgpt-user.json

• OAI-AdsBot

  Job

  Validate the safety of pages submitted as ads on ChatGPT

  Follows robots.txt

  Not stated

  IP list

  openai.com/adsbot.json

Full headers, as documented:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Note that OAI-SearchBot sends a full desktop Chrome preamble and the others do not. A rule that blocks "headless-looking" agents can catch three and miss the one that decides ChatGPT search. OpenAI says a robots.txt change can take about 24 hours to reach its systems.

What are Anthropic's, Perplexity's and Mistral's tokens?

• Anthropic

  Token

  ClaudeBot

  Job

  Collect content that could contribute to model training

  robots.txt

  Honoured, including Crawl-delay

• Anthropic

  Token

  Claude-SearchBot

  Job

  Improve search result quality

  robots.txt

  Honoured

• Anthropic

  Token

  Claude-User

  Job

  Fetch pages when people ask Claude questions

  robots.txt

  Honoured

• Perplexity

  Token

  PerplexityBot

  Job

  Surface and link websites in Perplexity results

  robots.txt

  Honoured; Perplexity recommends allowing it

• Perplexity

  Token

  Perplexity-User

  Job

  Visit a page when a user's question needs it

  robots.txt

  Generally ignored, as the fetch is user-requested

• Mistral

  Token

  MistralAI-User

  Job

  Visit pages for user actions in its assistant

  robots.txt

  Governs user-initiated requests

• Mistral

  Token

  MistralAI-Index

  Job

  Automated indexing, not used for training

  robots.txt

  Not stated separately

• Mistral

  Token

  MistralAI-Training

  Job

  Build training datasets

  robots.txt

  Can be disallowed

Anthropic publishes one IP file for its bots at claude.com/crawling/bots.json and warns that blocking its IPs may stop it reading your robots.txt. Perplexity publishes one file per agent and says changes can take up to 24 hours to show. Perplexity's documented strings are:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

What are Google's and Apple's tokens?

Both use one crawler for search and a separate robots.txt token, with no crawler behind it, for AI training.

• Googlebot

  Is it a crawler?

  Yes

  What it controls

  Google Search, including all its search features

• Google-Extended

  Is it a crawler?

  No user agent of its own

  What it controls

  Whether crawled content is used for Gemini training and grounding; Google says it does not affect Search inclusion or ranking

• Applebot

  Is it a crawler?

  Yes

  What it controls

  Search in Spotlight, Siri and Safari; its data may also help train Apple's models

• Applebot-Extended

  Is it a crawler?

  No, it does not crawl

  What it controls

  Opt-out from training Apple's foundation models; disallowed pages can still appear in search

So you will never see Google-Extended or Applebot-Extended in a log. They exist only as names in robots.txt, read by the main crawler.

What are Meta's, Amazon's and Common Crawl's tokens?

• meta-externalagent

  Operator

  Meta

  Job

  Training foundation models or indexing content for products

  Notes

  Sent as meta-externalagent/1.1

• Meta-ExternalFetcher

  Operator

  Meta

  Job

  Fetch links at a user's request

  Notes

  May bypass robots.txt

• facebookexternalhit

  Operator

  Meta

  Job

  Fetch pages shared in Meta's apps for previews

  Notes

  Not an AI crawler

• Amazonbot

  Operator

  Amazon

  Job

  Improve products; may train Amazon AI models

  Notes

  Does not support crawl-delay

• Amzn-SearchBot

  Operator

  Amazon

  Job

  Search experiences; not used for generative training

  Notes

• Amzn-User

  Operator

  Amazon

  Job

  User actions such as Alexa queries; not used for generative training

  Notes

• CCBot

  Operator

  Common Crawl

  Job

  An open web corpus that many model builders use

  Notes

  Sent as CCBot/2.0 (https://commoncrawl.org/faq/)

Common Crawl says plainly that it is aware of crawlers falsely identifying themselves as CCBot, and publishes IP ranges so you can check.

Bytespider, the third-busiest agent on our site, has no operator documentation we could cite, so we list it without a description.

Which tokens are easiest to confuse?

Five pairs cause most of the expensive mistakes, because each pair shares an operator and does opposite jobs.

GPTBot and OAI-SearchBot. Both are OpenAI. GPTBot is for training; OAI-SearchBot decides whether you can appear in ChatGPT's search answers. A rule written for one does nothing to the other. A site that means to opt out of training but writes User-agent: OAI-SearchBot removes itself from ChatGPT search instead.

ClaudeBot and Claude-User. ClaudeBot collects training data. Claude-User fetches a page because a person just asked Claude about it. Blocking the second, Anthropic says, may reduce your visibility in user-directed search.

Googlebot and Google-Extended. Googlebot is the crawler for Google Search, and Google's AI features draw on its index. Google-Extended is only a word in robots.txt. Disallowing it opts content out of Gemini training and grounding and, by Google's statement, does not touch Search.

Applebot and Applebot-Extended. The same split at Apple: Applebot crawls for search, Applebot-Extended is a training opt-out that does not crawl at all.

meta-externalagent and facebookexternalhit. The first is Meta's training and indexing crawler, the busiest agent on our own site. The second fetches a page to build a preview when someone shares your link in Meta's apps. Blocking the second breaks your link previews and has nothing to do with AI.

How do you filter logs for AI crawlers?

Search the User-Agent field for each token as a case-insensitive substring, then group the hits by token, not by operator. Grouping by operator hides the job: OpenAI's 7,060 requests to our site were mostly its search agent, with a smaller share from its training agent and a few user-triggered fetches, and each of those needs a different decision. Keep the list of tokens in one place, such as a small file your log queries read, and update it when you re-read the operators' pages. A list that lives inside a dashboard query is a list nobody updates.

How are tokens matched against robots.txt groups?

By the product token, without regard to case. RFC 9309, the robots.txt standard, says crawlers must match their product token against each group's user-agent lines case-insensitively, and must use the most specific group that matches. If no group names the crawler, it uses the wildcard group.

Three consequences are worth knowing. A group can list several user-agent lines, so you can apply one rule set to GPTBot, ClaudeBot and CCBot together. A crawler that finds a group naming it ignores the wildcard group, so rules you put only under * do not apply to an agent you also named. And a token you never named, such as a new agent an operator adds, gets your wildcard rules, which is exactly what happened to the Mistral agents on our site.

How should you use these tokens in practice?

Four rules cover nearly every case.

1. One robots.txt group per token you care about. Do not rely on the wildcard to express a decision about a named agent. The robots.txt chapter has the syntax and the precedence rules.
2. Match logs on the token as a case-insensitive substring, without the version. meta-externalagent arrives in lower case; most others do not.
3. Verify before you act. Before blocking a surprising agent or celebrating a new one, check the request's IP against the operator's published list.
4. Re-read the operators' pages every quarter. Ours drifted in a few months. Each operator's page is linked in the sources below and in each agent's entry in the crawler directory.

The next chapter, training versus search crawlers, explains what allowing or blocking each of these jobs actually costs you.