How to verify an AI crawler is genuine

How do I verify a crawler is genuine?

Two methods, and which applies depends on the operator. Google and Apple document reverse DNS: look up the hostname of the accessing address, check its suffix, then resolve that hostname forward and confirm it returns the same address. OpenAI, Anthropic, Perplexity and Amazon publish machine-readable files listing the addresses their agents fetch from. Common Crawl documents both, which makes CCBot the one agent you can check either way.[^ccbot-verification]

Why the header proves nothing

User-Agent is a request header. The client chooses its value, nothing validates it, and copying a documented crawler string takes one line of code.

There is a specific reason AI crawler tokens are the ones most worth forging: sites make exceptions for them. A rate limiter that lets GPTBot through, a bot ruleset that whitelists PerplexityBot, a paywall that serves full text to OAI-SearchBot — every one of those is an incentive to claim the name. Common Crawl says this outright, warning that crawlers falsely identifying themselves as CCBot exist.[^ccbot-spoofing]

So "GPTBot fetched 4,000 pages last week" is not a measurement until it is verified. It is a count of requests that said they were GPTBot.

Method one: reverse DNS

Google documents the canonical four-step procedure, and it is worth following exactly rather than approximately.[^google-rdns-procedure]

1. Run a reverse DNS lookup on the accessing IP address from your logs.
2. Verify the returned domain name is googlebot.com, google.com or

googleusercontent.com.

1. Run a forward DNS lookup on that domain name.
2. Verify the result is the same as the original accessing IP address.

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Step four is the whole point. Reverse DNS records are controlled by whoever holds the address block, so anyone with an IP range can publish a PTR record that ends in a plausible-looking hostname. Only the round trip — hostname resolves back to the same address — proves that the operator controls both halves. A verification routine that stops after step two is a verification routine that accepts forgeries.

Apple documents the same shape of check against its own suffix: the reverse hostname for genuine Applebot traffic ends in applebot.apple.com.[^applebot-rdns] Common Crawl documents the same shape again, with genuine CCBot addresses resolving under crawl.commoncrawl.org.[^ccbot-verification]

Two failure modes to handle. A reverse lookup can return nothing at all, which is not evidence of forgery so much as absence of evidence — treat it as unverified rather than fake. And DNS resolution costs a round trip per unique address, so cache results and verify unique IPs rather than every log line.

Method two: published IP range files

The other operators publish the addresses themselves, as documents you fetch.

OpenAI's per-agent split is the most useful arrangement here, because it lets you identify which of the four agents made a request without trusting the header at all.[^openai-ip-json] Anthropic publishes one combined file for ClaudeBot, Claude-User and Claude-SearchBot,[^anthropic-bots-json] so with Anthropic you can prove the traffic is theirs but must use the header to tell which agent it was. Perplexity publishes one file per agent.[^perplexity-ip-json] Amazon publishes a page of addresses.[^amazonbot-ip-page] Common Crawl publishes the same ranges it documents for reverse DNS as JSON.[^ccbot-verification]

Fetch the file; do not copy from it. Operators add and retire ranges without announcing a schedule, and a hardcoded list fails in the direction that hurts: genuine crawler traffic silently reclassified as fake, and whatever you built on that classification quietly wrong. Refresh on a schedule, and treat a failed refresh as a reason to fall back to the last known list rather than to reject everything.

The rule to verify before you block

Verification and access control are different jobs, and conflating them causes problems.

Anthropic makes this explicit: it states that IP blocking may not work correctly for its agents, and recommends robots.txt instead.[^anthropic-bots-json] The published file exists so you can confirm traffic is theirs, not so you can build a firewall rule from it. The same caution applies generally — an address-based blocklist built from a snapshot of a published range is a blocklist that starts being wrong immediately and never tells you.

Use the published data to answer "was this real?" Use robots.txt to answer "may it fetch this?" Those are the tools each mechanism is documented for.

A workable verification routine

For a site that wants trustworthy crawler numbers rather than a research project, this is enough.

1. Filter log lines by the documented product tokens, not by version, and not

by the full header.

1. Extract unique source addresses for each token over the window you care

about. This is typically a small set, which is what makes the rest cheap.

1. For Google, Apple and Common Crawl, run the reverse-then-forward check on

each unique address, and cache the verdict.

1. **For OpenAI, Anthropic, Perplexity, Amazon and Common Crawl, fetch the

published ranges** on a schedule and test membership.

1. Report verified and unverified separately. Never merge them into one

number. The unverified column is itself a useful signal: a sudden spike in requests claiming a crawler token, from addresses that do not verify, is scraping wearing a costume.

1. Do not act on an unverified count. If a page appears to be heavily

crawled by an agent that never verifies, the honest conclusion is that you do not know it was crawled at all.

Step five is where most log analysis goes wrong, because merging the columns turns a measurement into a flattering estimate. Attensira's crawler logs exist to run this routine continuously rather than as an occasional audit, and any log pipeline can be made to do the same — what matters is that the verified and unverified counts stay apart.

Where verification breaks in practice

Four situations account for most of the cases where a correct routine produces wrong answers, and all four are about your infrastructure rather than the crawler.

The address in your log is not the client's. Behind a CDN, load balancer or reverse proxy, the origin sees the proxy's address. Verification has to run against the forwarded client address, and only if you trust the layer that set it — a forwarded header from an untrusted source is as forgeable as a user agent. Getting this wrong produces a total verification failure that looks like an epidemic of forgery.

Logs at the edge and logs at the origin disagree. A CDN that blocks a crawler at the edge produces no origin log line at all, so the origin's view is "this crawler never visited" when the truth is "this crawler was turned away". The absence looks identical to never having been discovered, and the fix is in a completely different place.

Addresses are recycled. A cached verification verdict is a verdict about an address at a moment. Cache the result for hours, not months, and re-verify periodically rather than treating a pass as permanent.

IPv6 and IPv4 need the same treatment. A routine that verifies one and silently passes or fails the other will misclassify a growing share of traffic, and the direction of the error depends on which default you chose without noticing.

What a verified log actually answers

The reason to do any of this is that a verified crawler log answers questions nothing else can, and each of them is directly actionable.

The fourth row is the one that most often surprises people, because the site's configuration and the site's behaviour disagree: robots.txt permits the crawler, and something in front of the origin rejects it anyway. Without verification you cannot tell that case apart from a scraper being correctly blocked, which is why the two columns have to stay separate.

What verification cannot tell you

It confirms who fetched, and nothing beyond that. A verified OAI-SearchBot request proves OpenAI's search crawler read the page. It does not prove the page entered an index, was retrieved for any question, or was ever cited. Those are the steps no operator exposes, and the reason the pillar is careful to describe crawling as a precondition for citation rather than a cause of it.

There is also a limit on the other side. Verification tells you an agent belonged to an operator; it says nothing about what that operator subsequently did with the response. A verified training crawler fetch and a verified search crawler fetch look identical in your logs, and the difference between them lives entirely in the token and the operator's own description of it — which is why the user-agent reference and this chapter are worth reading as a pair rather than separately.

Next: llms.txt, the file people publish hoping to influence the step verification cannot see.

[^google-rdns-procedure]: Google Search Central, verifying Googlebot, retrieved 2026-09-03. [^openai-ip-json]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^anthropic-bots-json]: Anthropic support documentation, retrieved 2026-09-03. [^perplexity-ip-json]: Perplexity crawler documentation, retrieved 2026-09-03. [^applebot-rdns]: Apple support, About Applebot, retrieved 2026-09-03. [^amazonbot-ip-page]: Amazon developer documentation, Amazonbot, retrieved 2026-09-03. [^ccbot-spoofing]: Common Crawl, CCBot, retrieved 2026-09-03. [^ccbot-verification]: Common Crawl, CCBot, retrieved 2026-09-03.