How AI assistants find your pages

Who actually came looking for our pages last month?

Here is the real list for our own site. Attensira counts crawler requests to attensira.com in edge middleware, the only place a crawler visit can be seen, because AI crawlers do not run JavaScript and never trigger a browser analytics script. The table at the top of this page counts how many different pages each agent fetched in the 30 days to 4 October 2026, read over the Attensira MCP that morning. Two things in it set up the rest of this chapter.

The biggest visitor feeds none of the answers we track. Meta's agent made about nine in ten of all AI crawler requests. Its job, in Meta's words, is training models or improving products by indexing content. It is not the agent behind any answer on ChatGPT, Claude, Perplexity or Google.

The agents that do feed answers are quieter and spread wider. PerplexityBot made 595 requests and spread them over more than half as many different pages; Meta's agent made over a hundred thousand and stayed on a few hundred more. Most of OpenAI's requests came from its search agent, not its training agent, as the training versus search chapter shows. These counts are by user agent, which any client can claim, so we read them as evidence of coverage, not as a verified log.

The volume of crawling tells you very little. Which agent visited, and which job it does, tells you a lot.

What are the two ways an assistant finds a page?

Every AI assistant that cites the web reaches a page by one of two routes.

Ahead of time, through an index. A search agent crawls the web continuously and builds an index. When someone asks a question, the assistant searches that index, picks pages, reads passages and cites some of them. Your page has to be in the index before the question is asked. OAI-SearchBot, Claude-SearchBot and PerplexityBot do this job for their products. Google's AI features use Google's own search index, built by Googlebot.

On demand, for one question. When a person asks something that needs a specific page, or pastes a link, the assistant can fetch that page there and then. ChatGPT-User, Claude-User and Perplexity-User do this. They are the visits that come closest to "someone is asking about you right now".

A third kind of agent, the training crawler, does neither. GPTBot, ClaudeBot and Meta-ExternalAgent collect pages that may be used to train future models. A model trained on your page may know about you, but it will not link to you from that knowledge. Training is about what the model remembers; indexing and fetching are about what it can cite.

Which agents does each assistant use?

Here is each operator's own description, fetched on 4 October 2026.

ChatGPT (OpenAI). OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. ChatGPT-User is used for certain user actions in ChatGPT and custom GPTs, and OpenAI notes robots.txt rules may not apply to those user-initiated requests. GPTBot is used to make its generative AI foundation models more useful and safe. OpenAI says a robots.txt change can take about 24 hours to reach its systems.

Claude (Anthropic). Claude-SearchBot navigates the web to improve search result quality. Claude-User accesses websites when individuals ask Claude questions. ClaudeBot collects content that could contribute to model training. Anthropic says blocking Claude-User or Claude-SearchBot may reduce your site's visibility in user-directed search.

Perplexity. PerplexityBot is designed to surface and link websites in Perplexity's search results, and Perplexity recommends allowing it. Perplexity-User visits a page when a user's question needs it and generally ignores robots.txt, because a person requested the fetch. Changes may take up to 24 hours to show.

Google AI Overviews and AI Mode. There is no separate AI crawler. A page must be indexed and eligible to appear in Google Search with a snippet to be a supporting link, and Google says there are no additional requirements. Googlebot indexes it. Google-Extended is a robots.txt token with no user agent of its own; it controls whether content is used for Gemini training and grounding and, in Google's words, does not affect inclusion or ranking in Search.

The full user-agent strings and robots.txt tokens are in the AI crawler guide's user-agent chapter, and each agent has its own page in the crawler directory.

How does a fetched page become part of an answer?

In five steps, and the crawler only sees the first.

1. Fetch. A search agent requests the page, or a user agent fetches it for a question. If robots.txt blocks it, the server errors, or the content needs JavaScript to appear, the chain stops here.
2. Index or read. A search agent stores the page in its product's index. A user agent reads it once.
3. Retrieve. A person asks a question. The assistant searches for pages that might answer it, often running several related searches. Google calls this query fan-out: issuing multiple related searches across subtopics and sources to build one response.
4. Select. The model picks passages from the retrieved pages.
5. Cite. The product attaches some of those pages as sources, and may or may not name the brand behind them.

Our own pages show how much falls away between step one and step five. The PerplexityBot and OpenAI agents fetched hundreds of our pages in the month. Our AI crawler guide's pillar page was attached to none of the 4,518 answers we had read on ChatGPT and Perplexity in the 30 days to 14:45 UTC on 4 October 2026, when we re-read the page counts. That afternoon count is larger than the 4,374 answers our measurement guides report from the morning of the same day. Our pricing page was attached 72 times, counting both addresses engines use for it (58 citations as www.attensira.com/pricing, 14 as attensira.com/pricing). Being fetched made both pages eligible. Only one was selected, and that one mostly as background to answers that named other brands.

What happens if you block each kind of agent?

The consequences differ by job, so block by job, never with a wildcard.

• A training agent (GPTBot, ClaudeBot)

  What you lose, by the operator's description

  Future pages left out of that operator's training

  What you keep

  Search and user-triggered access, so today's answers can still cite you

• A search agent (OAI-SearchBot, Claude-SearchBot, PerplexityBot)

  What you lose, by the operator's description

  Inclusion in that product's search answers

  What you keep

  Nothing that helps you be cited there

• A user agent (ChatGPT-User, Claude-User)

  What you lose, by the operator's description

  The assistant reading your page when a person asks about it

  What you keep

  Training and indexing, if allowed

• Googlebot

  What you lose, by the operator's description

  Google Search, and with it AI Overviews and AI Mode

  What you keep

  Nothing in Google

• Google-Extended (a token)

  What you lose, by the operator's description

  Use of your content for Gemini training and grounding

  What you keep

  Google Search inclusion and ranking, by Google's statement

The commonest expensive mistake is a single rule that disallows every user agent, written to stop training. It also removes you from every AI search product that honours robots.txt. The what to allow and what to block chapter of the crawler guide works through the trade-offs for different kinds of site.

Some user agents, Perplexity-User among them, are documented as generally ignoring robots.txt because a person asked for the page. If you truly need to stop a fetch, robots.txt is the wrong tool; access control on the server is the right one. For most sites the question never comes up, because a fetch for someone's question is exactly the visit that can get you cited.

How do you check what the assistants can see?

Four checks, in order of how often they find a problem.

1. Read your robots.txt as each agent. For each search and user agent above, confirm the paths you want cited are allowed. Remember that groups for different agents are separate: a rule for GPTBot does nothing to OAI-SearchBot.
2. Fetch the page without a browser. Request it with a plain HTTP client and read what comes back. If the paragraph that answers the question is missing, an agent that reads the initial HTML cannot use it either.
3. Read your server or edge logs by agent. Not your browser analytics, which have never seen a crawler. Group by agent and job, then ask which sections the search agents reach and which they never do.
4. Check the answers, not just the visits. A page that search agents fetch but answers never cite has a selection problem, not an access problem. That is the subject of the next chapter, on ranking versus being cited.

The crawler URL inspector shows how each agent's robots.txt rules apply to a given URL, and the page token inspector shows the text a crawler actually receives.

What does this mean for where to start?

If your pages cannot be fetched by the search agents, nothing else in this guide can help yet, so check access first. It is also the cheapest fix there is: a robots.txt edit, a rendering change, a redirect cleaned up.

Once access is clean, stop looking at crawler volume. The number of requests says more about how aggressive a training crawler is than about your visibility. Watch two things instead: whether the search agents reach the sections you care about, and whether the answers to your buyers' questions cite those sections. On our own site, the first is fine and the second is the work.