Crawling and indexing

How much of a site actually gets crawled?

Less than you assume, even on a busy site. In the 30 days to 4 October 2026, attensira.com received 143,601 requests from AI crawlers, read over the Attensira MCP. The widest crawler, Meta-ExternalAgent, reached 873 pages; OpenAI's three agents together reached 428, under half as many. And at least 121 pages were not fetched by a single AI crawler in the month.

The unfetched pages are not random. They include /api-platform and 18 pages under it, the /tools/bot-access-score page, and about 60 programmatic blog posts on agency and industry variants of one topic. Many are variants of a single template. A page no crawler has fetched does not exist to that engine's answers, however good it is.

Request volume says little about coverage. A crawler that re-fetches your homepage thousands of times does nothing for a page three links deep. The counts are also labelled by the user agent each request claimed. None was IP-verified, so some may be other bots using a crawler's name.

What is crawling?

A bot requesting a page over HTTP and downloading it. Crawlers discover URLs from links, from XML sitemaps and from push protocols such as IndexNow. Before fetching, a well-behaved crawler reads your robots.txt. RFC 9309, the robots.txt standard, says crawlers should not cache the file for more than 24 hours and must treat an unreachable robots.txt that returns a server error as "disallow everything".

That last rule bites more sites than it should: a robots.txt that returns 500 during an outage tells compliant crawlers to stop.

What is indexing?

What happens after the fetch. The search engine parses the page, renders it if needed, picks a canonical URL among duplicates, extracts the text and links, decides whether the page is worth keeping, and stores it so it can be retrieved later. A page can be crawled and not indexed: duplicates, thin pages, soft 404s and pages marked noindex are all fetched and then dropped.

Indexing is also where AI search starts. Google states that a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. AI assistants with their own search, ChatGPT, Perplexity and Claude, build their own indexes with their own crawlers.

How do AI crawlers differ from search crawlers?

• Several bots per company, with different jobs. OpenAI, Anthropic and Perplexity separate training crawlers from search crawlers and from fetchers a person triggers. Each has its own robots.txt rule. See the crawler directory.
• Most do not run JavaScript. Attensira's own documentation on AI traffic notes that AI crawlers do not execute JavaScript. Text that appears only after scripts run can be invisible to them. Serve content in the HTML.
• User-triggered fetches may ignore robots.txt. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked.

How do you control crawling and indexing?

• Stop a crawler fetching a path

  Use

  robots.txt Disallow

  Note

  the URL can still be indexed from links elsewhere

• Keep a page out of the index

  Use

  noindex meta tag or X-Robots-Tag header

  Note

  the page must be crawlable so the tag is seen

• Consolidate duplicates

  Use

  redirect or rel=canonical

  Note

  hints, not commands

• Get new pages found fast

  Use

  sitemap, internal links, IndexNow

  Note

  discovery, not a guarantee

• See what Google indexed

  Use

  Search Console page indexing report

  Note

  Google only

What are the objections about crawling and indexing?

"Block a page in robots.txt to deindex it." That stops the crawler seeing your noindex. Allow the crawl and serve noindex.

"If Google has us, the AI engines do too." Each AI search product runs its own crawler and index, so coverage in one says nothing about the others. AI search shows which of them fetched us.

"Crawl budget is our problem." Google's crawl budget guidance is aimed at very large or very fast-changing sites. Most sites need clean links and a sitemap, not budget tuning.

"Our analytics would show AI crawlers." Client-side analytics run in the browser; crawlers do not. Only server or edge logs see them.

How does Attensira track crawling?

Attensira's crawler logs record which AI and search crawlers fetched which of your pages, how often and for which job, captured at the edge where bots are visible. Its crew fixes what blocks crawling, such as robots.txt rules, missing sitemap entries or content hidden behind scripts, and ships each fix after your approval. Crawler logs shows the same coverage reading for your own site, per crawler, including the pages no crawler fetched.