AI crawler logs analysis tool: What to use and what the log tells you

What the log should report, in order

1. Identity. Which named crawler made the request, and whether it is really that crawler. A user-agent string is free to type; an IP range is not. Verify against the operator's published ranges before you trust the visit. Our verifying a crawler guide walks through the check.
2. Coverage. Which of your pages were fetched. A page that should answer a tracked question and was never fetched is a different problem from one that was fetched and passed over.
3. Outcome. Status codes per crawler. A wall of 403s is your own robots.txt or WAF talking, not a content problem.
4. Volume. Requests per crawler over time, and which sections of the site get the attention.
5. Spoofing. Requests claiming to be a crawler from outside its published ranges. Good tools flag these separately so they do not inflate real coverage.

The distinction that changes the report

Training crawlers and answer crawlers are not the same visit. OpenAI documents GPTBot for training and OAI-SearchBot for surfacing websites in ChatGPT search features. PerplexityBot indexes for Perplexity search results; Perplexity-User fetches when a person asks a question.

Cloudflare's AI crawler reference separates the two categories, and a log analysis should too. Being crawled for training does not mean you will be quoted in ChatGPT or Perplexity next week. When you present results to a stakeholder, show search crawlers on their own row.

On attensira.com, the 30 days from 5 September to 4 October 2026 recorded 143,601 crawler requests read over the Attensira MCP on 4 October 2026. Training traffic dominated the total count; PerplexityBot still touched 331 distinct URLs with 621 requests, and OpenAI's crawlers touched 428 URLs with 7,536 requests. Those coverage numbers are the input to citation work, not proof of citations.

How do you run a log pass without a data team?

1. Export 30 days from your CDN or host.
2. Filter to AI user agents you care about for search: at minimum OAI-SearchBot and PerplexityBot if ChatGPT and Perplexity matter to you.
3. Pivot by path and day; mark your twenty priority URLs.
4. Join status codes; anything other than 200 on a priority URL is a finding.
5. Compare before and after robots.txt or WAF changes. OpenAI notes robots.txt adjustments can take about 24 hours to reflect in its systems.
6. Archive a monthly summary so you can see drift without keeping every raw line forever.

For filter lists, user-agent strings and robots.txt patterns, see the AI crawler guide.

What should you do when the log looks healthy but answers ignore you?

That is the fetched-not-cited case. The crawler read the page and another source won the question. Fix the passage, not the server:

• Lead with a forty- to sixty-word block that answers the exact question.
• Put facts that change often beside a date and source.
• Remove contradictions between the title, the meta description and the first paragraph.

Then re-read the same questions on ChatGPT and Perplexity with the sample size shown. Monitoring alone does not change an answer; shipping the page change does.

Where Attensira fits

Attensira reads crawler traffic at the edge rather than inferring it from a sampled answer. The AI crawler and bot directory lists the agents and their tokens, the AI crawler URL inspector shows what a single URL returns to a bot, and the AI crawler guide covers verifying that a visit is genuine. Crawler logs on your site join to the answers the product reads daily, so "never fetched" and "fetched, never cited" show up as different inbox items.

When a page that should answer a tracked question has never been fetched, that is the finding that starts the fix.

What no log can tell you

A crawl is not a citation. Crawler logs prove access and nothing more. ChatGPT and Perplexity select a small set of sources per question, and being fetchable is a precondition, not a promise. Report crawl coverage and citation separately, or you will read a healthy log as a win.

That separation is why this category of tool belongs next to answer tracking, not instead of it. Use logs to unblock access; use answers to judge whether the page you shipped is the one engines quote.

How do you evaluate a log tool before you buy?

Ask five questions on a demo call or in a trial export:

1. Which log sources are supported? Edge-only products miss origin-only sites; desktop parsers struggle when logs live in object storage.
2. Does it split training and search crawlers by default? If everything lands in one bucket, you will misread training noise as search progress.
3. Can it verify IPs? Without verification, block lists and allow lists built from the report are unsafe.
4. What retention do you get? Thirty days is enough to debug a robots.txt change; ninety days or rollups help before-and-after studies on rewrites.
5. Can you export URL-level history? Dashboards are useful; CSV or API access lets you join crawl dates to publish dates in your CMS.

None of those questions mention price tiers. They predict whether the tool answers "can ChatGPT and Perplexity search crawlers reach this URL?" which is the question logs are good at.

What does a weekly review ritual look like?

Pick five URLs that should win buyer questions this quarter. Each Monday, filter last week's log lines for OAI-SearchBot and PerplexityBot on those paths only. Record last fetch time, fetch count and worst status code. If last fetch predates your last deploy, the engine has not seen the new copy yet, so pause answer comparisons until a fetch lands.

Once a month, widen the filter to your entire /blog/ or /docs/ tree and look for sections with zero search-crawler hits. Those deserts are usually discovery problems: missing internal links, sitemap gaps or navigation rendered only in JavaScript. Fix discovery before you rewrite passages on pages crawlers never open.

Share the monthly summary with content and engineering in one slide: green rows for URLs fetched after the last change, amber for fetched but stale, red for never fetched or blocked. That color coding keeps the conversation on access, where logs are authoritative, instead of on citations, where logs are silent.

What belongs in a handoff to content?

When logs show repeated 200 responses on a URL but answers never cite it, attach three artifacts to the rewrite ticket: the log snippet with dates, the Perplexity or ChatGPT answer screenshot or export with sources highlighted, and the paragraph the engine quoted from the competitor instead. Writers then see the shape of passage that won, not just a metric that says "0 citations."

When logs show 403 or 401 responses, hand engineering the exact path, user agent and timestamp. WAF rules often block by path prefix; a fix that opens /blog/ but not /docs/ shows up immediately in the next week's filter.

What should you ignore in the log?

Human browsers, monitoring probes you run yourself, and ancient SEO crawlers unless you still optimize for classic search. Some teams filter to ten AI tokens and miss Googlebot; if Google Search still matters, keep traditional search crawlers in a secondary tab.

Ignore single-day spikes from training crawlers unless you are negotiating crawl budget with a vendor. They rarely explain next-day citation changes on buyer questions in ChatGPT or Perplexity.