# AI crawler log analysis: which bots fetch your pages, and which they skip

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> An AI crawler log analysis tool pulls GPTBot, PerplexityBot and other agents out of server or CDN logs. Attensira also reads crawlers at the edge. A crawl is access, not a citation.

- Canonical page: https://attensira.com/features/crawler-logs
- Source: converted from this page's own HTML. It carries no claim
  records, because the page is not a content-pipeline entity.

An AI crawler log analysis tool reads your server or CDN access logs and reports which AI crawlers fetched which pages, how often, and what status they got back. Attensira’s crawler logs do the same at your edge, where JavaScript never runs, so you can separate crawl coverage from citation rates on the questions you track.

Traffic · AI crawlers attensira.com · 30 days

Requests · pages fetched

- Meta · Meta-ExternalAgent Training: 126,349 · 866 pages
- OpenAI · GPTBot, OAI-SearchBot, ChatGPT-User 86% search index, 11% training, 3% live answer: 7,465 · 428 pages
- ByteDance · Bytespider Training: 3,344 · 121 pages
- Perplexity · PerplexityBot Search index: 621 · 331 pages
- Anthropic · ClaudeBot, Claude-User 66% search index, 34% live answer: 103 · 48 pages
- Google AI Overview, Gemini No visits seen in the window: 0

Agents are as each request named itself; none is presented as verified.

![Analyst agent](https://attensira.com/agents/analyst-64.webp) Analyst attensira.com workspace, read over the Attensira MCP · 4 Oct 2026 · 30 days

- ![Analyst agent](https://attensira.com/agents/analyst-64.webp) Analyst · Reads which pages bots fetch, and flags tracked questions whose pages no crawler reaches.
- ![Researcher agent](https://attensira.com/agents/researcher-64.webp) Researcher · Turns pages bots fetch with no prompt behind them into questions worth tracking.

## What should a log analysis report, in order?

Whether you use a one-off log parser or continuous edge logging, the same fields answer whether AI bots can reach the pages you care about.

1. ### Identity
   
   Which named crawler made the request, verified against the operator’s published IP ranges — a user-agent string alone is easy to spoof.
2. ### Coverage
   
   Which URLs were fetched. A page that should answer a tracked question and was never fetched is a different problem from one that was fetched and passed over.
3. ### Outcome
   
   Status codes per crawler. A wall of 403 responses is usually robots.txt or a WAF, not a content-quality problem.
4. ### Volume and spoofing
   
   Requests over time, by site section, plus hits from addresses outside the operator’s published ranges.

## What does one line of the log say?

When the request came, which agent it named, which page it asked for and what the site answered. A page that answers a crawler with a 403 or a redirect is worth checking first: robots rules, a login wall or a moved URL.

## Which log analysis tools are worth knowing?

These are the tools answer engines name when buyers ask for AI crawler log analysis. Each reads a different surface; pick the one that matches where your traffic is logged.

| Tool | What it reads | Best for |  |
| --- | --- | --- | --- |
| Cloudflare AI Crawl Control | Cloudflare AI Crawl Control | Cloudflare edge logs | Sites already on Cloudflare; separates training crawlers from answer crawlers |
| Crawl Lab | Crawl Lab | Apache or Nginx logs, parsed in the browser | A first look without uploading the log file |
| Screaming Frog Log File Analyser | Screaming Frog Log File Analyser | Large log files, desktop | Big datasets and URL-level crawl analysis |
| LogLens | LogLens | Cloudflare, Apache and Nginx logs | Continuous monitoring with crawler verification |
| Attensira crawler logs | Attensira | Your edge middleware | Ongoing AI traffic beside prompt tracking, with pages no bot reached flagged |

Crawler log analysis, at a glance

- Part of: Measure
- Approval: Draft by default; claims wait in every mode [Inbox](https://attensira.com/features/inbox)
- Available: Growth and Business. Raw requests are kept 90 days; a daily rollup with counts only is kept indefinitely.
- Plans: 50, 150 or 350 prompts, 300, 600, 1,200 credits a month; Enterprise above [Pricing](https://attensira.com/pricing)
- In the docs: [AI traffic](https://docs.attensira.com/traffic/ai-traffic)

Checked 4 Oct 2026

What Attensira keeps

- Raw requests: 90 daysEach request as it arrived: when, path, agent named.
- Daily rollup: IndefinitelyCounts per day, page and agent; no IP or referrer stored.

## What do crawler logs not prove?

Training crawlers and answer crawlers are not the same visit. GPTBot and ClaudeBot often collect for training; OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch so an answer can cite a source. Being crawled for training does not mean you will be quoted.

- A crawl is not a citation. Report crawl coverage and citation rates separately, or a healthy log reads like a win when answers still ignore you.
- User agents are self-reported and easy to spoof unless you verify IP ranges.
- Nothing appears until a crawler fetches a page; on a low-traffic site that can take days.
- Browser analytics never sees AI crawlers; only server or edge logging can.

## Why did each crawler fetch the site?

Volume alone does not say which fetches can lead to an answer. The traffic log labels each request by why that user agent fetches: training, search indexing or a live answer.

- of OpenAI's requests were search indexing: 86%
- n=7,465 requests
- of Anthropic's requests fetched for a live answer: 34%
- n=103 requests
- of Meta-ExternalAgent's requests were training: 100%
- n=126,349 requests

![Analyst agent](https://attensira.com/agents/analyst-64.webp) Analyst · Labelled every request attensira.com AI traffic log, read over the Attensira MCP on 4 Oct 2026 · labelled by the user agent each request claimed · 4 Sep 2026 to 4 Oct 2026

Software that does your AI-search visibility work, with you in control of every change.

## What else do people ask about crawler logs?

### What is an AI crawler log analysis tool?

It classifies server or edge requests by AI crawler user agent, counts which paths were fetched, and often labels why the bot came. It is not citation tracking: a fetch does not mean an answer will cite you. Attensira keeps raw requests 90 days and a count-only daily rollup after that.

### What is the best AI crawler logs analysis tool?

It depends where your traffic is logged. Cloudflare AI Crawl Control fits Cloudflare sites; Crawl Lab parses Apache or Nginx logs locally; Screaming Frog suits large one-off exports; LogLens and similar tools monitor continuously. Attensira fits teams that want edge logging beside daily answer readings on the same prompts.

### Why does my analytics show no AI crawler traffic?

Because browser analytics runs JavaScript in the visitor's browser, and AI crawlers do not run JavaScript. A zero there describes the tool, not the crawlers. Crawler logs count requests in your edge middleware instead.

### Which plans include crawler logs?

Growth and Business. Starter does not include crawler logs; every other measurement on this site works without them.

### Does more crawling mean more citations?

No. Crawler counts and citation counts are not two views of one number, and a rise in one does not predict the other. Read crawling as whether your pages can be read at all, and citation rate as whether they are used.

### What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

They are OpenAI agents that come for different reasons: training, building a search index, and fetching a page live because someone asked ChatGPT right now. Crawler logs report the reason beside each.

### Is any personal data kept?

Requests are measured at your own edge, on your own traffic. Raw requests are deleted after 90 days, and the rollup kept after that holds counts only, with nothing that identifies a person.

### Related features

- [Citation tracking Then cited?](https://attensira.com/features/citation-tracking)
- [Model surfaces Not crawlers](https://attensira.com/features/models)
- [Prompt tracking Suggestions](https://attensira.com/features/prompt-tracking)
- [AI bots directory Each crawler](https://attensira.com/bots)
- [AI crawler guide Read logs](https://attensira.com/guides/ai-crawler-guide)
- [Crawler URL inspector One URL](https://attensira.com/tools/crawler-url-inspector)
- [AI crawler log analysis (blog) Tools](https://attensira.com/blog/analyzing-log-files)
- [Did the change work? Before and after](https://attensira.com/features/attribution)

### In the docs

- [AI traffic Docs](https://docs.attensira.com/traffic/ai-traffic)
- [Pages Docs](https://docs.attensira.com/traffic/pages)

## See which bots read you, and which pages they never reach.