The AI crawler guide: bots, jobs and robots.txt

Here are the ones that visited ours. Attensira counts crawler requests to attensira.com in edge middleware, which sees every request before a page renders. Server and CDN logs see those fetches too; a browser analytics script never does, because AI crawlers do not run JavaScript. These are the 30 days to 4 October 2026, read over the Attensira MCP that morning, 120,143 requests in all.

• Meta

  Agents seen

  Meta-ExternalAgent

  Requests

  108,487

  Pages

  847

• OpenAI

  Agents seen

  OAI-SearchBot, GPTBot, ChatGPT-User

  Requests

  7,060

  Pages

  419

• ByteDance

  Agents seen

  Bytespider

  Requests

  3,205

  Pages

  115

• Perplexity

  Agents seen

  PerplexityBot

  Requests

  595

  Pages

  331

• Microsoft

  Agents seen

  bingbot

  Requests

  515

  Pages

  203

• Google

  Agents seen

  Googlebot

  Requests

  182

  Pages

  79

• Anthropic

  Agents seen

  ClaudeBot, Claude-User

  Requests

  99

  Pages

  46

What each agent is for, in its operator's words, is in the sections below; ByteDance publishes no documentation for Bytespider that we could cite. These counts are by user agent, which any client can claim. Attensira reports them as directional evidence of coverage and does not present them as verified visits.

Three things in the table matter more than the totals.

Volume and value are unrelated. Meta's training agent made about nine of every ten AI crawler requests to our site, fetching each of its 847 pages 128 times on average; the frame at the top of this page divides every row the same way. The agents that feed answers people read, OAI-SearchBot and PerplexityBot among them, made a small fraction. A dashboard that ranks crawlers by requests puts the least useful one on top.

One operator, several jobs. Of OpenAI's 7,060 requests, Attensira classified about 85 in 100 as search indexing, 12 in 100 as training and 3 in 100 as live fetches for a user's question, by user agent. Those are three different agents with three different robots.txt tokens, and blocking one does not block the others.

Some agents you would expect are absent by design. There is no Gemini or AI Overviews row. Google's AI features are fed by Googlebot, and Google-Extended, the token that controls Gemini training and grounding, has no user-agent string of its own. It exists only as a rule in robots.txt.

An AI crawler is a bot an AI company runs to fetch web pages. The word covers several different things, and the difference is the whole subject of this guide. Some agents collect pages to train future models. Some index pages so an AI search product can retrieve and cite them. Some fetch one page, once, because a person just asked an assistant a question that needs it. Each has its own name, its own robots.txt token, and its own cost if you block it.

Most confusion about AI crawlers comes from treating them as one thing: "block AI bots" or "allow AI bots". Neither instruction means anything until you say which job you are allowing or blocking.

Nearly every documented agent does one of four jobs.

• Training

  What it does

  Collects pages that may be used to train future models

  Examples from operators' documentation

  GPTBot, ClaudeBot, Meta-ExternalAgent; Google-Extended and Applebot-Extended as opt-out tokens; CCBot for Common Crawl's open dataset

  What blocking it costs

  Your future content is left out of that operator's training. No direct effect on being cited today

• Search indexing

  What it does

  Builds the index an AI search product retrieves from

  Examples from operators' documentation

  OAI-SearchBot, Claude-SearchBot, PerplexityBot; Googlebot and bingbot for the classic engines AI features sit on

  What blocking it costs

  You drop out of that product's answers. OpenAI says sites that disable OAI-SearchBot are not shown in ChatGPT search answers

• User-triggered fetches

  What it does

  Fetches one page because a person asked a question

  Examples from operators' documentation

  ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher

  What blocking it costs

  The assistant cannot read your page when a person asks about it. Some of these agents are documented as able to ignore robots.txt

• Product operations

  What it does

  Previews, safety checks, ads and verification

  Examples from operators' documentation

  Link-preview fetchers and ad crawlers

  What blocking it costs

  Usually unrelated to AI answers

The chapter on training versus search crawlers goes through each job and the documented consequences in detail.

The major operators each document more than one agent. In their own words, summarised:

• OpenAI: GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot surfaces websites in ChatGPT's search features. ChatGPT-User visits a page for certain user actions in ChatGPT and custom GPTs. OpenAI also notes that a robots.txt change can take about 24 hours to take effect in its systems.
• Anthropic: ClaudeBot collects content that could contribute to model training. Claude-SearchBot improves search result quality. Claude-User fetches a page when a person asks Claude a question. Anthropic notes that blocking Claude-User or Claude-SearchBot can reduce your visibility in user-directed search.
• Perplexity: PerplexityBot surfaces and links websites in Perplexity results and is not used to train foundation models. Perplexity-User fetches a page for a user's question and generally ignores robots.txt.
• Google: Googlebot crawls for Google Search, which includes AI Overviews and AI Mode. Google-Extended is a robots.txt token, not a crawler, controlling whether content may be used for Gemini training and grounding; Google says it does not affect inclusion or ranking in Search.
• Apple: Applebot powers search across Spotlight, Siri and Safari. Applebot-Extended is an opt-out token for training Apple's foundation models and does not crawl.
• Meta: Meta-ExternalAgent is for training foundation models or indexing content for products. Meta-ExternalFetcher fetches links at a user's request and may bypass robots.txt.
• Amazon: Amazonbot improves Amazon's products and services, and its content may be used to train Amazon's models.
• Common Crawl: CCBot builds an open repository of web data that many model builders use.

The exact user-agent strings and tokens for each are in the chapter on user agents and robots.txt tokens, and every agent has its own page in the crawler directory.

In stages, and a crawler is only the first one.

1. Fetch. A search or user-triggered agent requests the page. If robots.txt blocks it, or the server returns an error, nothing happens next.
2. Index or read. A search agent adds the page to its product's index. A user-triggered agent reads it for one answer.
3. Retrieve. When a person asks a question, the product searches its index, or the classic search engine it uses, for pages that might answer it.
4. Select. The model picks passages from the retrieved pages to build its answer.
5. Cite. The product attaches some of those pages as sources.

Crawler logs see step one. Citations happen at step five. That is why the two numbers never reconcile, and why Attensira's own documentation tells you not to try: a page can be fetched constantly and never cited, or cited from something read long ago and never fetched again. Crawling is necessary for citation. It is nowhere near sufficient.

Four, in rough order of how often we see them.

Blocking everything under the wildcard. A rule that disallows all user agents removes you from every AI search product that respects robots.txt, along with the training crawlers you meant to block. If the goal is to opt out of training, name the training agents and leave the rest.

Blocking the search agent while meaning the training agent. A rule for GPTBot does not affect OAI-SearchBot. A rule for OAI-SearchBot does what OpenAI says: you are not shown in ChatGPT search answers. Check that each token in your file does the job you think it does.

Forgetting that groups combine. The robots.txt standard, RFC 9309, says that when several groups match a user agent, their rules are combined. A specific group for one agent does not replace a general group that also matches it.

Letting robots.txt fail. RFC 9309 says a crawler must assume complete disallow when robots.txt is unreachable because of a server or network error, but may crawl everything when the file returns a status in the 400 range. A robots.txt that times out under load can block every compliant crawler until it recovers.

The chapter on robots.txt directives has the syntax, and the robots.txt generator writes a file with each agent named.

No. The standard itself says its rules are not a form of access authorization: they are a request that compliant crawlers honour. The training and search-indexing agents of the major operators document that they follow it.

User-triggered fetchers are different. Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch, and Meta says Meta-ExternalFetcher may bypass it for the same reason. If you need to stop a fetch outright, robots.txt is the wrong tool; access control on the server is the right one. For most sites the question does not arise, because a fetch for a person's question is exactly the visit that can get you cited.

By checking where the request came from, not what it claims to be. A user-agent string is free text, and scrapers routinely borrow the names of well-known crawlers. Operators publish ways to verify: lists of IP ranges, or a reverse-DNS method. Google's, for example, is a reverse lookup on the requesting IP that must end in googlebot.com, google.com or googleusercontent.com, followed by a forward lookup that must return the same IP.

Our own table above is unverified for exactly this reason: it is counted by user agent, and Attensira says so rather than presenting it as an audited list of visitors. The chapter on verifying a crawler gives each operator's method, and the crawler URL inspector shows how each agent's rules apply to a specific URL on your site.

llms.txt is a proposal, first published in September 2024, for a Markdown file at a site's root that summarises the site for agents. It is open for community input and has real adoption among documentation publishers.

What it is not is a crawler control or a ranking signal anyone has documented. None of the operators' crawler documentation reviewed for this guide says it reads llms.txt, and Google says its AI features need no AI text file. Publishing one is cheap and harmless, and can help agents that look for it. It does not replace robots.txt, server-rendered pages or answering the question. The llms.txt chapter covers what to put in it, and the llms.txt generator drafts one.

At the server or the edge, never in browser analytics. Crawlers do not run JavaScript, so a browser-side script has never seen one; if your analytics shows no AI crawler traffic, that is a fact about the analytics. Read the raw request logs, or count requests in edge middleware, as Attensira does for its own site and for customers who install its AI traffic tracking.

When you read them, ask four questions:

1. Which agents visit, and which jobs do they do? Group by agent, then by job. A total is meaningless.
2. Which sections do the search agents reach? A section the search agents never fetch cannot become a fresh citation. Check it for robots rules, login walls and redirect loops.
3. Which pages do user-triggered agents fetch? Those are pages a person asked an assistant about. They are a free list of questions worth tracking.
4. Do the visits verify? Before acting on a surprising agent, check its IP against the operator's published method.

Because most of them read the HTML the server sends and stop there. Attensira's own measurement depends on this fact: crawlers do not run JavaScript, which is why their visits only show up at the server or the edge. The same fact applies to what they read. A page whose answer is assembled in the browser by a script can look complete to a person and nearly empty to a crawler.

Three patterns cause most of the trouble.

Content rendered only in the browser. If the paragraph that answers a question is fetched and inserted by JavaScript after the page loads, an agent that reads the initial HTML may never see it. Server-render or pre-render the text that matters. The test is simple: fetch the page with a plain HTTP client and read the response. If the answer is not in it, assume the crawlers cannot read it either.

Login walls and consent walls. A page behind a login is invisible to every crawler. A cookie or consent overlay that replaces the content in the HTML has the same effect. Keep public answers public, and make sure overlays sit on top of the content rather than instead of it.

Redirect chains and soft errors. A crawler following three redirects to reach a page, or receiving a page that says "not found" with a success status, wastes its budget and may index the wrong thing. Keep one hop from any old URL to its current page, and return real error codes.

The page token inspector shows what text a crawler actually receives from a URL, and the content structure score checks whether the answer sits where an extractor will find it.

Not instantly, and not at the same speed for every operator. OpenAI documents that it can take about 24 hours from a robots.txt update for its systems to adjust. Other operators fetch and cache robots.txt on their own schedules. Two practical consequences follow.

First, after fixing an accidental block, give it at least a day before checking whether the agent returns, and read your logs rather than assuming. Second, a block that was live for a week may have dropped pages from an AI search index, and getting them back depends on the agent recrawling them, which you cannot force. Re-reading the answers to the questions those pages serve, a week or two later, is how you find out whether they are back.

In what they feed and how much they fetch. Googlebot feeds Google Search, and Google says its AI features draw on the same index: a page must be indexed and snippet-eligible to appear as a supporting link in AI Overviews or AI Mode. There is no separate Google AI crawler to allow. If Googlebot can index the page, Google's AI features can use it, subject to snippet controls.

The AI-native operators run their own indexes, and they are smaller and younger than Google's. On our own site, Googlebot made 182 requests across 79 pages in the 30 days, against 595 across 331 pages for PerplexityBot and 7,060 across 419 pages for OpenAI's agents. The AI-native operators fetched our pages more widely than Google did. We draw no conclusion about ranking from that. What it does show is that each operator keeps its own copy of your site on its own schedule, which is one reason a fix can appear in one engine's answers days or weeks before another's, and why each engine has to be re-read separately after a change.

For a site that wants to be found in AI answers, the default that follows from the documentation is:

• Allow the search-indexing agents: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and of course Googlebot and bingbot.
• Allow the user-triggered agents: ChatGPT-User, Claude-User, Perplexity-User. They fetch pages people are asking about.
• Decide on the training agents: GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot, and the Google-Extended and Applebot-Extended tokens. This is a business decision about your content being used to train models. Blocking them does not, by the operators' own descriptions, remove you from today's answers.
• Keep robots.txt fast and reliable, because a failing file blocks every compliant crawler.

The last chapter, what to allow and what to block, works through the trade-offs for different kinds of site.

Attensira is an AI-native SEO agency run by software: the service of an SEO agency at the price of a tool. Crawler access is the first thing its Web engineer checks, because no other work matters if the engines cannot read the page. It reads your robots.txt against every documented agent, flags rules that block search or user-triggered agents, and drafts the fix as a change for you to approve. On plans with AI traffic, Attensira counts crawler requests at the edge, shows which pages each agent reaches, and turns pages that crawlers fetch with no tracked question behind them into suggested questions to track. The crawler logs feature shows the product side.

"We blocked AI bots to protect our content." Then check which ones. If the rule named the search and user-triggered agents too, you also opted out of being cited. Most sites that want to protect content from training mean the training agents only.

"The crawlers hammer our servers." Look at which agent. On our own site, one training agent made most of the requests. Rate-limiting or blocking that agent is a reasonable choice that does not touch the agents that feed answers.

"If a crawler fetched our page, we must be in the answers." No. Fetching is step one of five. Our crawler logs and our citation data are separate datasets on purpose, and they do not line up.

"robots.txt keeps scrapers out." It does not. It is a request that compliant crawlers honour. Anything that ignores it needs server-side controls.

Six chapters, each answering one question:

1. AI crawler user agents and robots.txt tokens: the exact strings and tokens.
2. Training versus search: whether blocking AI crawlers stops you being cited.
3. Robots.txt directives for AI crawlers: how to write the rules.
4. How to verify an AI crawler is genuine: checking that GPTBot is GPTBot.
5. llms.txt and where it stands: whether it is worth publishing.
6. What to allow and what to block: the decision for your site.

Every factual statement in the guide is listed with its source and the date it was read, at the foot of each page. Operators change their agents and their documentation, so the guide is reviewed every quarter, and each chapter shows when it last changed. If an operator's page says something different from ours, theirs is right and ours is the bug.