How to verify an AI crawler is real

What does verifying a crawler look like in practice?

Two commands and a list. Here are real lookups, run on 4 October 2026.

Reverse DNS, then forward DNS, for an IP claiming to be Googlebot:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The name ends in googlebot.com, and the forward lookup returns the same IP. That is Google's documented test, passed. The same reverse lookup on 17.241.219.1 returned 17-241-219-1.applebot.apple.com, the domain Apple documents for Applebot.

A published IP list, for an IP claiming to be GPTBot. OpenAI publishes the ranges each of its agents uses. We fetched openai.com/gptbot.json and checked two addresses: 132.196.86.5 falls inside one of its 18 ranges; 8.8.8.8 falls in none. A request from the second claiming to be GPTBot is not GPTBot.

That is all verification is: ignore what the request says it is, and check where it came from.

Now the uncomfortable part, at the top of this page: our own AI traffic board for attensira.com reports none of a month's AI crawler requests as verified. That is deliberate. Attensira counts visits by user agent at the edge and says plainly in its documentation that it does not claim to prove a request came from a named crawler. Every crawler table in this guide is therefore labelled as unverified. Before you block, allow or celebrate a specific agent on the strength of a log line, do the check above.

Why can't you trust a user-agent string?

Because the client writes it. A User-Agent header is free text. Any script can send GPTBot/1.4 or Googlebot/2.1, and scrapers routinely borrow well-known crawler names to get past filters that wave known bots through. Common Crawl says outright that it is aware of crawlers falsely identifying themselves as CCBot.

So a user-agent string tells you what a request claims. The IP address, which has to be real for the request to get a response, tells you where it came from. Every operator's verification method is built on the IP.

Which verification method does each operator support?

• Google

  Agents

  Googlebot and other Google crawlers

  Method

  Reverse DNS to googlebot.com, google.com or googleusercontent.com, then forward DNS; or JSON range files

  Where

  Google's verification page

• Apple

  Agents

  Applebot

  Method

  Reverse DNS in applebot.apple.com; or a JSON file of CIDRs

  Where

  Apple's Applebot page

• OpenAI

  Agents

  GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot

  Method

  One JSON range file per agent

  Where

  openai.com/gptbot.json and siblings

• Anthropic

  Agents

  ClaudeBot, Claude-SearchBot, Claude-User

  Method

  One JSON range file for all its bots

  Where

  claude.com/crawling/bots.json

• Perplexity

  Agents

  PerplexityBot, Perplexity-User

  Method

  One JSON file per agent

  Where

  perplexity.com/perplexitybot.json, perplexity-user.json

• Common Crawl

  Agents

  CCBot

  Method

  Reverse DNS, or a JSON range file

  Where

  index.commoncrawl.org/ccbot.json

• Amazon

  Agents

  Amazonbot

  Method

  A published list of IP addresses

  Where

  developer.amazon.com/amazonbot/ip-addresses

Two details matter. OpenAI's files are per agent, so you can check not just that a request is from OpenAI but which of its agents sent it, independent of the header. Anthropic's single file tells you a request is from Anthropic but not which bot.

Why do fake crawlers borrow AI names at all?

Because many sites wave known crawlers through. A firewall rule, a rate limiter or a paywall that exempts any request whose header says GPTBot or Googlebot is an open door for anyone who types those words. Scrapers that would be throttled as unknown clients get unlimited access by claiming to be a crawler the site has decided to welcome.

The fix is to allowlist by address, never by header. Meta's documentation makes the same point about its own crawlers: you can allowlist either their user-agent strings or their IP addresses, and it calls the IP route more secure. The same holds for every operator. A rule that says "let GPTBot in" should mean "let in requests from the ranges in gptbot.json", and the header becomes a label you check against the address, not a key.

How do user-triggered fetchers change verification?

They come from more places, and their lists are bigger and change faster.

A search crawler runs from a fleet the operator controls and schedules. A user-triggered fetcher runs whenever a person somewhere asks a question, so it needs capacity close to where people are. That shows in the files: on 4 October 2026, OpenAI's list for ChatGPT-User held 230 ranges, against 18 for GPTBot and 39 for OAI-SearchBot, and it had been regenerated nine days earlier. A stale copy of that file is the one most likely to reject real visitors.

Two operators complicate things further. Meta documents that Meta-ExternalFetcher may bypass robots.txt, and its crawler page offers no published list we could find for it. Perplexity's user-agent file listed only 4 ranges. For agents like these, decide in advance how you treat a request you cannot verify, and write the decision down.

How big are the lists, and how often do they change?

Big enough to need a script, and they change often enough that you should fetch them on a schedule. As fetched on 4 October 2026:

• openai.com/gptbot.json

  Ranges listed

  18

  Generated

  22 September 2026

• openai.com/searchbot.json

  Ranges listed

  39

  Generated

• openai.com/chatgpt-user.json

  Ranges listed

  230

  Generated

  25 September 2026

• claude.com/crawling/bots.json

  Ranges listed

  28

  Generated

  2 October 2026

• perplexity.com/perplexitybot.json

  Ranges listed

  8

  Generated

• perplexity.com/perplexity-user.json

  Ranges listed

  4

  Generated

• index.commoncrawl.org/ccbot.json

  Ranges listed

  5

  Generated

• Google common-crawlers.json

  Ranges listed

  317

  Generated

  2 October 2026

Three of these files had been regenerated in the two weeks before we fetched them. A copy you downloaded last quarter will reject real requests and accept nothing new. Fetch daily, cache for a day at most, and keep the last good copy in case a fetch fails.

How do you verify at scale?

Not one request at a time. A practical setup:

1. Fetch every operator's list daily into one table of ranges, labelled by operator and agent.
2. Match each crawler-looking request against the ranges for the agent it claims to be. A claim that does not match is spoofed.
3. Cache reverse DNS results for operators that use it, keyed by IP, for a day. Never run a lookup inline on every request.
4. Label your counts as verified, spoofed or unverifiable, and keep the three apart in reports.
5. Alert on a sudden change in the share of spoofed requests for one token. It usually means a scraper has started borrowing that name.

Many CDNs and bot-management products maintain verified-bot lists for you. If you use one, check which agents it covers; the AI operators add agents faster than most lists are updated.

How should you report crawler traffic you have not verified?

Honestly labelled, in three columns rather than one. Verified requests are evidence of what an operator's crawler did. Spoofed requests are evidence of a scraper, and belong in a security view, not a visibility one. Unverifiable requests, from operators that publish no list, are a claim you cannot check, and should be reported as a claim.

If you cannot verify at all, which is where most teams start, say so on the chart. Our own crawler tables in this guide carry the line "counted by user agent, not verified" for exactly this reason. A request count that mixes all three is still useful as a rough measure of coverage: which sections get fetched, and how often. It is not useful for deciding that one operator values your site more than another, and it should never be presented as an audited record of who visited.

The label costs nothing, and it stops a spoofer's traffic being read as a sign that an AI company has taken an interest in you.

What should you do with a request that fails verification?

Treat it as what it is: an unknown client using a false name. Three reasonable responses, depending on the load it puts on you:

• Ignore it in your analytics. Do not count it as the crawler it claims to be.
• Rate-limit it like any other unknown client.
• Block it at the firewall if it is heavy or hostile.

What you should not do is change robots.txt because of it. A spoofer does not read robots.txt, and a rule aimed at the name it borrowed will only affect the real crawler.

Is it ever a mistake to verify by blocking IPs?

Yes, when the goal is an opt-out. Anthropic warns that blocking its IP addresses may stop it reading your robots.txt, so an IP block may not reliably opt you out of its crawling. If you want an operator's real crawler to stay away, say so in robots.txt to its token. Use IP rules for traffic you want gone entirely, and use verification to tell the two apart.

What can verification not tell you?

Two things.

What a verified crawler will do with the page. A real OAI-SearchBot fetch tells you the page can be in ChatGPT's search index. It does not tell you the page will be retrieved for any question, or cited. That is the subject of the AI search visibility guide.

Whether an unverifiable agent is honest. Some agents, such as Bytespider on our own site, come from operators that publish no documentation we could cite. You cannot verify them, only decide how much unverified traffic you are willing to serve.

The next chapter, on llms.txt, turns from who is reading your site to what you hand them when they arrive.