Google-Site-Verification: allow it or block it?

Why should Google-Site-Verification always be allowed?

Because blocking it hurts only you. Google documents Google Site Verifier as the fetcher that fetches Search Console verification tokens¹. When you prove you own a property by uploading an HTML file to your root or adding a meta tag to your homepage, this agent reads it. It fetches one URL, looks for one string and leaves.

There is no trade-off. Block it and verification fails. Without a verified property there is no Search Console: no index coverage report, no performance data, no manual action notices and no sitemap submission. Google loses nothing. You lose the only first-party view you have of how Google sees your site, often including the data you would use to diagnose whatever made you write the rule.

Why does it ignore robots.txt, and why is that right?

Google classifies it as a user-triggered fetcher, a fetcher initiated by users to perform a fetching function within a Google product, and uses Site Verifier acting on a user's request as its own example of the class⁵. Google states that its user-triggered fetchers generally ignore robots.txt rules³.

Here that is clearly correct. The fetch happens because you asked for it, from your own Google account, about your own site. Making it obey a file on that site would create a loop: a mistake in robots.txt could stop you gaining the access you need to diagnose the mistake in robots.txt.

So the Allow group in the robots.txt section above is a record, not a mechanism. Google is unlikely to consult it. A colleague writing a blanket rule six months from now will, and a line saying this fetcher is deliberately permitted is worth having in the file.

Why does verification fail when nothing is blocked?

Because robots.txt is almost never the cause. The verifier reads a response body, and anything that puts something else in that body breaks it silently:

• a CDN bot-management rule that serves a JavaScript challenge to unrecognised user agents;
• a WAF rule that treats a non-browser header as suspicious;
• a geographic or network block that returns an interstitial;
• a platform migration where the verification file or meta tag did not come across.

All four can return 200 OK with a page that does not contain your token, which from Google's side looks exactly like a site that never had one. Check what the fetcher actually receives, not what your browser receives.

The last case deserves emphasis. Verification is re-checked over time, not granted once. A property verified three years ago can quietly lapse the week you change hosts. If you are migrating, carry the verification file and meta tag across before you switch DNS.

How do you build a safe edge exception for it?

On verified addresses, never on the header. Google publishes the IP ranges for its user-triggered fetchers in user-triggered-fetchers.json and user-triggered-fetchers-google.json, and documents reverse DNS hostname patterns under googleusercontent.com and google.com for the class⁴.

The naive exception is dangerous. Google publishes the header as Google-Site-Verification/1.0²: short, guessable and unauthenticated. A rule that skips bot challenges for anything sending it is an open door with a Google-shaped label. Key the exception on the range file instead.

Note that the ranges and hostname patterns cover the user-triggered class, not this agent alone. A verified address tells you Google made the request; the token tells you which fetcher, once the address has held up.

What does normal Google-Site-Verification traffic look like?

Very little of it. A handful of hits at verification time and occasional re-checks afterwards is the normal shape. A sustained stream is not this agent behaving oddly; it is something else wearing the name, and the address check is how you find out. Match the bare token Google-Site-Verification, case-insensitively, since the 1.0 is a version Google can increment.

• Google-Site-Verification

  What it reads

  Your verification file or meta tag

  If you block it

  Search Console verification fails

• Google-InspectionTool

  What it reads

  A page you test in URL Inspection

  If you block it

  Your own tests fail

• Googlebot

  What it reads

  Pages for the search index

  If you block it

  You leave Search

What can Attensira show you about Google-Site-Verification?

AI traffic, on Growth and Business, groups fetches by operator and declared purpose at the edge, so a verification fetch never inflates a crawler total. It reports user agents as sent; the range-file check stays yours. Attensira also connects to Search Console where you grant access, and acts only within the access you grant. See crawler logs and the robots.txt generator.