For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Bots

Site verification fetchers

Verification bots confirm that you own a site. Google-Site-Verification fetches your Search Console token, the HTML file or meta tag you add to prove ownership. Google classes it as a user-triggered fetcher that generally ignores robots.txt. Blocking it breaks those two methods, not a DNS TXT record, so this is one agent where blocking only hurts the owner.

Start freeBook a demo

7 days free · card required

Google-Site-Verification · documentedGoogle
Operator
Google
What the fetch is for
User-triggered fetch
robots.txt
May not apply
Verify a visit by
IP ranges + reverse DNS
attensira.com/robots.txt
No named group
Requests to attensira.com, 30 days
0 filed as Google-Site-Verification
Researcher agentResearcherGoogle documentation, our AI traffic board (n=143,601) and robots.txt ·

Which verification agents are documented?

Fetches the token you placed on the site to prove you own it. Blocking it breaks only your own HTML-file and meta-tag verification; a DNS TXT record needs no fetch.

robots.txtGoogle-Site-Verification

Allow it explicitly. Google states user-triggered fetchers generally ignore robots.txt anyway, but an Allow group records the decision so that a later blanket rule does not get written over the top of it.

User-agent: Google-Site-VerificationAllow: /
Researcher agentResearcherRule per Google documentation ·

Why did verification fail when the file is there?

Usually an edge rule, not robots.txt. A CDN bot-management setting, a WAF challenge or a geographic block can return a challenge page instead of your file, and the verifier reads the response body, so the substitution breaks verification silently. Google's user-triggered fetcher ranges are what to allowlist.

Without a verified property there is no Search Console: no index coverage, no performance data, no manual action notices.

VerificationGoogle-Site-Verification
Reverse DNS ends in
*.googleusercontent.com*.google.com
IP range file
user-triggered-fetchers-google.json

The range file and reverse DNS patterns cover Google's user-triggered fetchers as a class, not this agent alone, so a verified address establishes Google as the source and the token identifies which fetcher within that class. The Google-Site-Verification entry, and the method step by step: how to verify a crawler.

Researcher agentResearcherTranscribed from Google documentation ·

Why write an Allow group for a fetcher that generally ignores robots.txt?

To record the decision. Google states its user-triggered fetchers generally ignore robots.txt, so the group changes nothing on the wire today, but it stops a later blanket rule being written over the top of it by someone who did not know the agent was there.

What else do people ask?

What is a verification bot?
A fetcher that reads a token you placed on your site to prove you own it. Google-Site-Verification is the documented example: it fetches the HTML file or meta tag Search Console asks you to add.
Does Google-Site-Verification obey robots.txt?
No. Google classifies it as a user-triggered fetcher and states that these generally ignore robots.txt, because the fetch follows a specific action a person took.
What happens if I block Google-Site-Verification?
The HTML-file and meta-tag methods fail, and a property that relies on them loses Search Console: index coverage, performance data and manual action notices. A DNS TXT record verifies with no fetch, so it is unaffected. Nobody else is affected, which makes it one of the few agents where blocking only harms the site owner.
How do I verify a Google-Site-Verification request?
Match the address against Google's user-triggered-fetchers.json and user-triggered-fetchers-google.json, or check reverse DNS for a hostname under googleusercontent.com or google.com with a forward lookup back.

Where do these facts come from?

One claim on this page, with the page it was read from; last checked .

  1. Google documents Google Site Verifier as the user-triggered fetcher that fetches Search Console verification tokens.

    developers.google.com · retrieved

    “fetches Search Console verification tokens”

Want whatever it takes to win the answer?

Start freeBook a demo

7 days free · card required