For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Start free

AI crawler log analysis: which bots fetch your pages, and which they skip

An AI crawler log analysis tool reads your server or CDN access logs and reports which AI crawlers fetched which pages, how often, and what status they got back. Attensira’s crawler logs do the same at your edge, where JavaScript never runs, so you can separate crawl coverage from citation rates on the questions you track.

Start freeSee what AI says about you

7 days free · card required

Traffic · AI crawlersattensira.com · 30 days
Requests · pages fetched
Meta · Meta-ExternalAgentTraining
126,349 · 866 pages
OpenAI · GPTBot, OAI-SearchBot, ChatGPT-User86% search index, 11% training, 3% live answer
7,465 · 428 pages
ByteDance · BytespiderTraining
3,344 · 121 pages
Perplexity · PerplexityBotSearch index
621 · 331 pages
Anthropic · ClaudeBot, Claude-User66% search index, 34% live answer
103 · 48 pages
Google AI Overview, GeminiNo visits seen in the window
0

Agents are as each request named itself; none is presented as verified.

Analyst agentAnalystattensira.com workspace, read over the Attensira MCP · · 30 days

  • Analyst agentAnalyst · Reads which pages bots fetch, and flags tracked questions whose pages no crawler reaches.

  • Researcher agentResearcher · Turns pages bots fetch with no prompt behind them into questions worth tracking.

What should a log analysis report, in order?

Whether you use a one-off log parser or continuous edge logging, the same fields answer whether AI bots can reach the pages you care about.

  1. Identity

    Which named crawler made the request, verified against the operator’s published IP ranges — a user-agent string alone is easy to spoof.

  2. Coverage

    Which URLs were fetched. A page that should answer a tracked question and was never fetched is a different problem from one that was fetched and passed over.

  3. Outcome

    Status codes per crawler. A wall of 403 responses is usually robots.txt or a WAF, not a content-quality problem.

  4. Volume and spoofing

    Requests over time, by site section, plus hits from addresses outside the operator’s published ranges.

What does one line of the log say?

When the request came, which agent it named, which page it asked for and what the site answered. A page that answers a crawler with a 403 or a redirect is worth checking first: robots rules, a login wall or a moved URL.

AI traffic · ledgerly.com5 of 1,832 requests
to · labelled by the user agent each request claimed
TimeAgentPathStatus
OAI-SearchBot, operated by OpenAI/pricing200
ChatGPT-User, operated by OpenAI/compare/ledgerly-vs-invoicely200
ClaudeBot, operated by Anthropic/guides/vat-on-invoices403
PerplexityBot, operated by Perplexity/blog/invoice-numbering-rules200
GPTBot, operated by OpenAI/old-pricing301
Analyst agentAnalyst · Read the logExample · Ledgerly, a fictional brand

Which log analysis tools are worth knowing?

These are the tools answer engines name when buyers ask for AI crawler log analysis. Each reads a different surface; pick the one that matches where your traffic is logged.

Which log analysis tools are worth knowing?
ToolWhat it readsBest for
Cloudflare AI Crawl ControlCloudflare AI Crawl ControlCloudflare edge logsSites already on Cloudflare; separates training crawlers from answer crawlers
Crawl LabCrawl LabApache or Nginx logs, parsed in the browserA first look without uploading the log file
Screaming Frog Log File AnalyserScreaming Frog Log File AnalyserLarge log files, desktopBig datasets and URL-level crawl analysis
LogLensLogLensCloudflare, Apache and Nginx logsContinuous monitoring with crawler verification
Attensira crawler logsAttensiraYour edge middlewareOngoing AI traffic beside prompt tracking, with pages no bot reached flagged

What do crawler logs not prove?

Training crawlers and answer crawlers are not the same visit. GPTBot and ClaudeBot often collect for training; OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch so an answer can cite a source. Being crawled for training does not mean you will be quoted.

  • A crawl is not a citation. Report crawl coverage and citation rates separately, or a healthy log reads like a win when answers still ignore you.
  • User agents are self-reported and easy to spoof unless you verify IP ranges.
  • Nothing appears until a crawler fetches a page; on a low-traffic site that can take days.
  • Browser analytics never sees AI crawlers; only server or edge logging can.

Why did each crawler fetch the site?

Volume alone does not say which fetches can lead to an answer. The traffic log labels each request by why that user agent fetches: training, search indexing or a live answer.

of OpenAI's requests were search indexing
86%
n=7,465 requests
of Anthropic's requests fetched for a live answer
34%
n=103 requests
of Meta-ExternalAgent's requests were training
100%
n=126,349 requests
Analyst agentAnalyst · Labelled every requestattensira.com AI traffic log, read over the Attensira MCP on 4 Oct 2026 · labelled by the user agent each request claimed · to

What else do people ask about crawler logs?

What is an AI crawler log analysis tool?
It classifies server or edge requests by AI crawler user agent, counts which paths were fetched, and often labels why the bot came. It is not citation tracking: a fetch does not mean an answer will cite you. Attensira keeps raw requests 90 days and a count-only daily rollup after that.
What is the best AI crawler logs analysis tool?
It depends where your traffic is logged. Cloudflare AI Crawl Control fits Cloudflare sites; Crawl Lab parses Apache or Nginx logs locally; Screaming Frog suits large one-off exports; LogLens and similar tools monitor continuously. Attensira fits teams that want edge logging beside daily answer readings on the same prompts.
Why does my analytics show no AI crawler traffic?
Because browser analytics runs JavaScript in the visitor's browser, and AI crawlers do not run JavaScript. A zero there describes the tool, not the crawlers. Crawler logs count requests in your edge middleware instead.
Which plans include crawler logs?
Growth and Business. Starter does not include crawler logs; every other measurement on this site works without them.
Does more crawling mean more citations?
No. Crawler counts and citation counts are not two views of one number, and a rise in one does not predict the other. Read crawling as whether your pages can be read at all, and citation rate as whether they are used.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
They are OpenAI agents that come for different reasons: training, building a search index, and fetching a page live because someone asked ChatGPT right now. Crawler logs report the reason beside each.
Is any personal data kept?
Requests are measured at your own edge, on your own traffic. Raw requests are deleted after 90 days, and the rollup kept after that holds counts only, with nothing that identifies a person.

See which bots read you, and which pages they never reach.

Start freeSee what AI says about you

7 days free · card required