Google-Extended vs Googlebot: a control token and the crawler it rides on

Is Google-Extended a crawler?

No. Google calls Google-Extended a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Google also states that Google-Extended does not have a separate HTTP request user-agent string; crawling is done with existing Google user agents.

So nothing called Google-Extended ever visits your site. Googlebot and Google's other crawlers fetch your pages. Google-Extended is a rule, written in robots.txt syntax, that Google reads and applies to how those already-fetched pages may be used afterwards. That is why it can sit in a User-agent: line without ever appearing in a log.

What does Googlebot decide?

Almost everything about your presence in Google. Google describes Googlebot as its primary crawler, and its access affects Google Search, including Discover and all Search features, plus products such as Google Images, Google Video and Google News. AI Overviews and AI Mode are Search features. Google says that to be eligible as a supporting link in either, a page must be indexed and eligible to be shown in Google Search with a snippet, and that there are no additional requirements beyond that.

That sentence settles the most common confusion on this topic. Your eligibility for AI Overviews is decided by Googlebot's crawl and Google's index, plus the snippet controls. It is not decided by Google-Extended.

What does blocking each one cost?

Blocking Google-Extended costs nothing in Search, by Google's own statement: it does not impact a site's inclusion in Google Search and is not used as a ranking signal. What it changes is whether Google may use your pages to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. If your objection is to that use, it is the right line to write.

• Google-Extended

  Google Search

  Unaffected

  AI Overviews and AI Mode

  Unaffected

  Gemini training

  Opted out

  Gemini Apps and Vertex AI grounding

  Opted out

• Googlebot

  Google Search

  Removed

  AI Overviews and AI Mode

  Removed

  Gemini training

  Not stated by Google

  Gemini Apps and Vertex AI grounding

  Not stated by Google

Blocking Googlebot costs Google Search, and every feature built on it, AI Overviews and AI Mode included. For a business that wants to be found, it is almost never the right call. The exceptions are hosts that should not be found at all, and even then noindex or authentication is usually the cleaner control, because a page disallowed in robots.txt can still be indexed from links without its content.

What should your robots.txt say?

For a site that wants to stay in Google Search and its AI features but keep its pages out of Gemini training and grounding:

User-agent: Google-Extended
Disallow: /

Leave Googlebot alone, or allow it explicitly. If you want to limit how much of a page AI Overviews may use without leaving Search, the tool is not robots.txt at all, but the snippet controls in the page itself, for example a robots meta tag with max-snippet:160 in its content attribute.

Google lists nosnippet, data-nosnippet, max-snippet and noindex as the ways to limit inclusion in its AI features. Each works through Googlebot's index, which is why they behave predictably. For a whole site there is now a coarser switch: Search Console's "Search generative AI" setting, rolled out to all sites by 31 August 2026, excludes a site from AI Overviews, AI Mode and AI Overviews in Discover while leaving it in ordinary results. Google says the change generally takes a few days to apply.

What does grounding mean, and why is it in the token?

Grounding is the step where a model checks or supplements what it is about to say against content from Google's index, to improve factuality. Google's description of Google-Extended covers grounding in Gemini Apps and Vertex AI as well as training. That makes the token broader than a pure training opt-out: blocking it also asks Google not to use your pages to ground answers in those Gemini products.

That is a real cost for some sites, and it deserves to be weighed rather than ignored. If your buyers use the Gemini app to research your category, a Google-Extended block asks Google not to ground those answers on your pages. Google Search's own AI features are a different matter, because they are Search features and Google says the token does not affect Search. But the Gemini app is not Google Search, and the token does reach it. Decide with that in view.

What should an agency tell a client who asks?

Three sentences. Googlebot is how you are found in Google, including AI Overviews and AI Mode, so keep it. Google-Extended decides whether Google may train Gemini on your pages and ground Gemini app answers on them, and it has no effect on Search, so decide it on what you think about that use. If you want to limit what AI Overviews quote, we do it with snippet controls on the pages, not in robots.txt.

Most clients who ask are worried about one of two things, and separating them is most of the advice:

• Losing traffic to AI answers

  What kind of question it is

  A Search question

  The control

  Googlebot plus the snippet rules in the page

• Having content reused by Gemini

  What kind of question it is

  A reuse question

  The control

  Google-Extended in robots.txt

• Being quoted at length in AI Overviews

  What kind of question it is

  A Search question

  The control

  max-snippet or data-nosnippet

What did our logs show?

Attensira counts crawler requests at the edge on attensira.com, the way its AI traffic report does for customers on Growth and Business. In the 30 days to 4 October 2026, Googlebot made 189 requests across 80 pages, out of 143,601 AI-crawler requests. Our board has rows for Google Gemini and Google AI Overview agents, and both read 0, which is what Google's documentation predicts: Google's AI features use pages Google's existing crawlers already fetched. Of the 189 Googlebot requests, 180 fell in the final seven days, so treat the monthly total as a burst rather than a rate.

That has a practical consequence for measurement. You cannot learn from your logs whether Google used a page for Gemini grounding or for an AI Overview, because there is no separate fetch to see. What you can see is whether Googlebot is reading the pages you want considered. Whether an AI Overview then cites one of them is a question you answer by reading the answers, not the logs.

What are the limits of a Google-Extended opt-out?

It is forward-looking and specific. It governs future training of Gemini models and grounding in Gemini Apps and Vertex AI. It does not describe removal of content already used. It does not cover Search features, which use the index. And it applies only to Google; other labs have their own tokens, from GPTBot to ClaudeBot to Applebot-Extended, each with its own scope.

What should you check on your own site?

Fetch your live robots.txt and search for Googlebot and Google-Extended. Confirm Googlebot is allowed on every host you want in Search. Decide Google-Extended on its own question, training and grounding, and write the reason above it. Then look at whether Google's AI features cite you for the questions your buyers ask. That is decided by the index, the snippet controls and the quality of the page, which is where the work is.

Check the wildcard group too. A token with no group of its own falls back to User-agent: *, so a site whose wildcard disallows the root has, without anyone deciding it, also disallowed Google-Extended, and Googlebot as well unless Googlebot has a group of its own. That second case is the expensive one, and it usually arrives with a staging configuration copied to production. Comment each group with the question it answers, who decided and when, so a later edit is a conscious change rather than a guess.