# Google-Extended vs Googlebot: a control token and the crawler it rides on

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> Googlebot crawls for Google Search, AI Overviews and AI Mode; Google-Extended is a token that governs Gemini training and grounding. What blocking each does.

- Canonical page: https://attensira.com/compare/google-extended-vs-googlebot
- Family: compare
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=google-extended-vs-googlebot

**In short:** Googlebot is the crawler behind Google Search, including AI Overviews and AI Mode. Google-Extended is not a crawler: it is a robots.txt token that tells Google whether pages Googlebot already fetched may train Gemini models or ground Gemini Apps and Vertex AI. Blocking it does not affect Google Search. Blocking Googlebot removes you from Search.

**Readings, 5 Sep 2026 to 4 Oct 2026**

| Measure | Value |
| --- | --- |
| Googlebot requests to attensira.com, 180 of them in the last 7 days (n=143,601 AI-crawler requests) | 189 |
| distinct pages Googlebot fetched | 80 |
| requests labelled Google-Extended, a token that never fetches (n=143,601 AI-crawler requests) | 0 |

_AI-crawler requests to attensira.com, Attensira MCP, attensira.com workspace · 4 Oct 2026 · n=143,601_

## Verdict

A token and a crawler, not two crawlers. Googlebot fetches pages for Google Search and every Search feature, AI Overviews and AI Mode included. Google-Extended rides on those fetches and only governs Gemini training and grounding. Most sites keep Googlebot and decide Google-Extended on its own question.

## The readings on this page

### The readings at the top

attensira.com AI traffic log, read over the Attensira MCP; user agents self-reported, none IP-verified · 5 Sep 2026 to 4 Oct 2026

| Reading | Value | Sample |
| --- | --- | --- |
| Googlebot requests to attensira.com, 180 of them in the last 7 days | 189 | n=143,601 AI-crawler requests |
| distinct pages Googlebot fetched | 80 |  |
| requests labelled Google-Extended, a token that never fetches | 0 | n=143,601 AI-crawler requests |

### Which of our pages did Googlebot fetch?

Of the eight attensira.com pages AI crawlers read most in the 30 days to 4 October 2026, Googlebot fetched three, and six of the top 30. Google-Extended never appears in a log at all. It is a rule Googlebot reads, not an agent that visits.

attensira.com AI traffic log, page rows of the 30 most-fetched pages, Attensira MCP, attensira.com workspace · 5 Sep 2026 to 4 Oct 2026 · labelled by the user agent each request claimed · n=8 of the 30 most-fetched pages

Google-Extended: A robots.txt token; never in a log.

| Our page | Google-Extended | Googlebot |
| --- | --- | --- |
| /contact-sales | never in a log | Not seen |
| / | never in a log | Fetched |
| /pricing | never in a log | Fetched |
| /about | never in a log | Not seen |
| /features/prompt-tracking | never in a log | Not seen |
| /features/content-optimization | never in a log | Not seen |
| /features/crawler-logs | never in a log | Fetched |
| /mcp | never in a log | Not seen |

### How do large sites write rules for Googlebot?

robots.txt of the top-10,000 Tranco domains (list K9QPW), read by Attensira · 3 Sep 2026 to 3 Sep 2026

| Reading | Value | Sample |
| --- | --- | --- |
| never mention Googlebot (7,326) | 91.2% | n=8,036 domains |
| allow Googlebot by name (554) | 6.9% | n=8,036 domains |
| disallow Googlebot at the root (156) | 1.9% | n=8,036 domains |

## How do Google-Extended and Googlebot compare?

Each cell cites a claim in the Sources section below.

| Criterion | Google-Extended | Googlebot |
| --- | --- | --- |
| **What it is** |  |  |
| What it governs | Gemini training and grounding, and Vertex AI | Search, Discover, Images, Video and News |
| Appears in your server logs | No · Never | Yes · Under its own user agent |
| Does it fetch pages? | No · It issues no requests | Yes · It is the crawler |
| robots.txt user-agent token | Google-Extended | Googlebot |
| **Effect on Search and AI Overviews** |  |  |
| Effect on Google Search inclusion | No · Google states it is not a ranking signal either | Yes · Total; no Googlebot, no Search listing |
| Documented robots.txt compliance | It is itself a robots.txt mechanism | Yes · Always respects robots.txt for automatic crawls |
| How to limit use in AI Overviews instead | Not this token; the Search Console generative AI setting does it site-wide | nosnippet, data-nosnippet, max-snippet or noindex on the page |
| Effect on AI Overviews and AI Mode | Not tied to them in Google's documentation | Supporting links must be indexed and snippet-eligible |
| **What we measured** |  |  |
| Top-10k domains disallowing it at the root (study, n=8,036) | 992 (12.3%) | 156 (1.9%) |
| Requests to attensira.com, 30 days to 4 Oct 2026 | None; it has no user agent | 189 requests, 80 pages |

## What is Google-Extended?

A robots.txt product token, not a crawler. It governs whether content Google has already crawled may train future Gemini models or be used for grounding in Gemini Apps and Vertex AI.

## What is Googlebot?

Google's primary web crawler. It fetches pages for Google Search and the surfaces built on the same index, including Discover, Images, Video and News.

## When should you choose Google-Extended?

Disallow Google-Extended when you do not want pages Google crawls used to train future Gemini models or to ground answers in Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal.

## When should you choose Googlebot?

Keep Googlebot allowed in nearly every case. It is the crawler behind Google Search and its features, and Google says a page must be indexed and snippet-eligible to appear as a supporting link in AI Overviews or AI Mode.

## When is neither the right choice?

If you want to stay in Search but out of AI Overviews and AI Mode, neither token is the tool: Google points to nosnippet, data-nosnippet, max-snippet and noindex, which work through Googlebot's index.

## Where does Attensira fit?

Worth a look if you want to see whether answers cite you after you decide: Attensira reads ChatGPT, Perplexity, Claude, Gemini and Google AI daily and counts Googlebot at your edge on Growth and Business. Whatever it takes to win the answer.

## What sets Google-Extended and Googlebot apart?

- Google-Extended is a usage rule applied to Googlebot's fetches, so disallowing it saves no crawl load: Googlebot keeps fetching the same pages for Search.
- The most common misreading runs in both directions: sites block Google-Extended expecting to leave AI Overviews, and others keep it allowed fearing they would leave them. Neither is what the token does.
- Grounding in the Gemini app is the cost people miss: blocking Google-Extended asks Google not to ground those answers on your pages, even though Search is untouched.

## Is Google-Extended a crawler?

No. Google calls Google-Extended a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Google also states that Google-Extended does not have a separate HTTP request user-agent string; crawling is done with existing Google user agents.

So nothing called Google-Extended ever visits your site. Googlebot and Google's other crawlers fetch your pages. Google-Extended is a rule, written in robots.txt syntax, that Google reads and applies to how those already-fetched pages may be used afterwards. That is why it can sit in a `User-agent:` line without ever appearing in a log.

## What does Googlebot decide?

Almost everything about your presence in Google. Google describes Googlebot as its primary crawler, and its access affects Google Search, including Discover and all Search features, plus products such as Google Images, Google Video and Google News. AI Overviews and AI Mode are Search features. Google says that to be eligible as a supporting link in either, a page must be indexed and eligible to be shown in Google Search with a snippet, and that there are no additional requirements beyond that.

That sentence settles the most common confusion on this topic. Your eligibility for AI Overviews is decided by Googlebot's crawl and Google's index, plus the snippet controls. It is not decided by Google-Extended.

## What does blocking each one cost?

Blocking Google-Extended costs nothing in Search, by Google's own statement: it does not impact a site's inclusion in Google Search and is not used as a ranking signal. What it changes is whether Google may use your pages to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. If your objection is to that use, it is the right line to write.

| If you disallow... | Google Search | AI Overviews and AI Mode | Gemini training | Gemini Apps and Vertex AI grounding |
| --- | --- | --- | --- | --- |
| Google-Extended | Unaffected | Unaffected | Opted out | Opted out |
| Googlebot | Removed | Removed | Not stated by Google | Not stated by Google |

Blocking Googlebot costs Google Search, and every feature built on it, AI Overviews and AI Mode included. For a business that wants to be found, it is almost never the right call. The exceptions are hosts that should not be found at all, and even then noindex or authentication is usually the cleaner control, because a page disallowed in robots.txt can still be indexed from links without its content.

## What should your robots.txt say?

For a site that wants to stay in Google Search and its AI features but keep its pages out of Gemini training and grounding:

```
User-agent: Google-Extended
Disallow: /
```

Leave Googlebot alone, or allow it explicitly. If you want to limit how much of a page AI Overviews may use without leaving Search, the tool is not robots.txt at all, but the snippet controls in the page itself, for example a robots meta tag with `max-snippet:160` in its content attribute.

Google lists nosnippet, data-nosnippet, max-snippet and noindex as the ways to limit inclusion in its AI features. Each works through Googlebot's index, which is why they behave predictably. For a whole site there is now a coarser switch: Search Console's "Search generative AI" setting, rolled out to all sites by 31 August 2026, excludes a site from AI Overviews, AI Mode and AI Overviews in Discover while leaving it in ordinary results. Google says the change generally takes a few days to apply.

## What does grounding mean, and why is it in the token?

Grounding is the step where a model checks or supplements what it is about to say against content from Google's index, to improve factuality. Google's description of Google-Extended covers grounding in Gemini Apps and Vertex AI as well as training. That makes the token broader than a pure training opt-out: blocking it also asks Google not to use your pages to ground answers in those Gemini products.

That is a real cost for some sites, and it deserves to be weighed rather than ignored. If your buyers use the Gemini app to research your category, a Google-Extended block asks Google not to ground those answers on your pages. Google Search's own AI features are a different matter, because they are Search features and Google says the token does not affect Search. But the Gemini app is not Google Search, and the token does reach it. Decide with that in view.

## What should an agency tell a client who asks?

Three sentences. Googlebot is how you are found in Google, including AI Overviews and AI Mode, so keep it. Google-Extended decides whether Google may train Gemini on your pages and ground Gemini app answers on them, and it has no effect on Search, so decide it on what you think about that use. If you want to limit what AI Overviews quote, we do it with snippet controls on the pages, not in robots.txt.

Most clients who ask are worried about one of two things, and separating them is most of the advice:

| The client's worry | What kind of question it is | The control |
| --- | --- | --- |
| Losing traffic to AI answers | A Search question | Googlebot plus the snippet rules in the page |
| Having content reused by Gemini | A reuse question | Google-Extended in robots.txt |
| Being quoted at length in AI Overviews | A Search question | max-snippet or data-nosnippet |

## Why do so many guides get this wrong?

Because the name invites it. "Google-Extended" sounds like a Google crawler for AI, and lists of AI user agents put it next to GPTBot and ClaudeBot as if it were the same kind of thing. The Perplexity answer we stored on 4 October 2026 for our tracked prompt "list of AI crawler user agents" opens its list of "AI crawler user agents" with GPTBot and OAI-SearchBot and names Google-Extended among them, as if it were something you could find in a log. Google's own page says it is a token with no user agent that governs training and grounding, not crawling or indexing.

The practical damage is in both directions. Some sites block Google-Extended believing it removes them from AI Overviews, and are surprised when it does not. Others avoid blocking it because they fear losing AI Overviews, which it was never going to affect. Both decisions rest on a misreading.

## How do large sites treat the two?

On 3 September 2026 we read the robots.txt files of the top 10,000 domains in a dated Tranco list and parsed 8,036 of them. 992 (12.3%) disallow Google-Extended at the root. Only 156 (1.9%) disallow Googlebot. Large sites, in other words, are already treating the training token and the search crawler as separate decisions, and are about six times as likely to write the first as the second. Most, though, have decided nothing in writing: 91.2% never mention Googlebot and 84.2% never mention Google-Extended, so both are governed by whatever the wildcard group says. Explicit allows are rarer still, 6.9% for Googlebot and 3.4% for Google-Extended.

The small Googlebot figure is worth a second look on your own site. A root-level Googlebot disallow on a commercial site is almost always a mistake left over from a staging configuration, a migration or a copy of someone else's file.

### How do large sites write rules for Googlebot?

Of 8,036 top-10,000 domains whose robots.txt we read on 3 September 2026, nine in ten never name Googlebot, so their wildcard group decides. Few allow it by name, and fewer still disallow it at the root, which on a commercial site is usually a leftover from staging.

**Readings, 3 Sep 2026**

| Measure | Value |
| --- | --- |
| never mention Googlebot (7,326) (n=8,036 domains) | 91.2% |
| allow Googlebot by name (554) (n=8,036 domains) | 6.9% |
| disallow Googlebot at the root (156) (n=8,036 domains) | 1.9% |

_Attensira robots.txt study, run of 3 September 2026 · 3 Sep 2026 · n=8,036_

## What did our logs show?

Attensira counts crawler requests at the edge on attensira.com, the way its AI traffic report does for customers on Growth and Business. In the 30 days to 4 October 2026, Googlebot made 189 requests across 80 pages, out of 143,601 AI-crawler requests. Our board has rows for Google Gemini and Google AI Overview agents, and both read 0, which is what Google's documentation predicts: Google's AI features use pages Google's existing crawlers already fetched. Of the 189 Googlebot requests, 180 fell in the final seven days, so treat the monthly total as a burst rather than a rate.

That has a practical consequence for measurement. You cannot learn from your logs whether Google used a page for Gemini grounding or for an AI Overview, because there is no separate fetch to see. What you can see is whether Googlebot is reading the pages you want considered. Whether an AI Overview then cites one of them is a question you answer by reading the answers, not the logs.

## What are the limits of a Google-Extended opt-out?

It is forward-looking and specific. It governs future training of Gemini models and grounding in Gemini Apps and Vertex AI. It does not describe removal of content already used. It does not cover Search features, which use the index. And it applies only to Google; other labs have their own tokens, from GPTBot to ClaudeBot to Applebot-Extended, each with its own scope.

## What should you check on your own site?

Fetch your live robots.txt and search for Googlebot and Google-Extended. Confirm Googlebot is allowed on every host you want in Search. Decide Google-Extended on its own question, training and grounding, and write the reason above it. Then look at whether Google's AI features cite you for the questions your buyers ask. That is decided by the index, the snippet controls and the quality of the page, which is where the work is.

Check the wildcard group too. A token with no group of its own falls back to `User-agent: *`, so a site whose wildcard disallows the root has, without anyone deciding it, also disallowed Google-Extended, and Googlebot as well unless Googlebot has a group of its own. That second case is the expensive one, and it usually arrives with a staging configuration copied to production. Comment each group with the question it answers, who decided and when, so a later edit is a conscious change rather than a guess.

## FAQ

### Does blocking Google-Extended remove me from AI Overviews?

Google's documentation does not tie Google-Extended to AI Overviews. It says Google-Extended governs training of Gemini models and grounding in Gemini Apps and Vertex AI, and does not affect inclusion in Google Search. AI Overviews and AI Mode are Search features whose supporting links must be indexed and snippet-eligible.

### Will Google-Extended show up in my server logs?

No. Google states that Google-Extended does not have a separate HTTP request user-agent string; crawling is done with existing Google user agents. You will see Googlebot fetch your pages. Google-Extended is a rule Google applies to how those fetched pages are used afterwards.

### How do I keep my pages out of AI Overviews but stay in Search?

Not with either token. For a whole site, Google's Search Console "Search generative AI" setting, rolled out to all sites by 31 August 2026, excludes it from AI Overviews, AI Mode and AI Overviews in Discover while ordinary results stay. For single pages or passages, use nosnippet, data-nosnippet or max-snippet, which Google also lists as AI-feature controls.

### Is blocking Googlebot ever sensible?

Rarely, for a site that wants customers. Googlebot decides whether you are in Google Search and every feature built on it. Staging hosts, private tools and pages that must never be found are the exceptions, and even then noindex or authentication is usually the better control.

### Does Google-Extended affect my rankings?

Google says it is not used as a ranking signal in Google Search and does not affect a site's inclusion in Search. If a ranking changes after you add the rule, look for another cause.

### How common is each block among large sites?

In our read of 8,036 top-10,000 domains on 3 September 2026, 992 (12.3%) disallowed Google-Extended at the root and 156 (1.9%) disallowed Googlebot. Large sites treat the training token and the search crawler as separate decisions, though most still name neither.

## Sources

Each entry below is one claim on this page. `retrieved` is the date the exporter or a human last saw the source say this, and nothing more: it is not a publication date and not a promise of re-verification. `cite` is the anchor on the canonical page where this claim is rendered.

### extended-not-a-crawler

Google states that Google-Extended does not have a separate HTTP request user-agent string, and that crawling is conducted using existing Google user agents.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-extended-not-a-crawler
- side: google-extended

> Google-Extended doesn't have a separate HTTP request user agent string
### extended-token

Google-Extended is used as a robots.txt user-agent token, described by Google as a standalone product token.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-extended-token
- side: google-extended

> Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models
### extended-controls

Google documents the Google-Extended token as controlling Gemini Apps training, training via the Vertex AI API for Gemini, and grounding in Gemini Apps and Vertex AI.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-extended-controls
- side: google-extended
### extended-no-search-impact

Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-extended-no-search-impact
- side: google-extended

> Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search
### extended-no-requests

Because Google-Extended issues no requests of its own, it never appears as a user agent in a server access log.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-extended-no-requests
- side: google-extended
### googlebot-purpose

Google documents Googlebot as the crawler whose access affects Google Search, Discover, Google Images, Video, News and related features.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-googlebot-purpose
- side: googlebot
### googlebot-token

Googlebot is controlled with the robots.txt user-agent token Googlebot.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-googlebot-token
- side: googlebot
### googlebot-search-impact

Googlebot access is what determines whether a page can appear in Google Search and the surfaces built on the same index.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-googlebot-search-impact
- side: googlebot
### googlebot-robots

Google describes its common crawlers, of which Googlebot is one, as always respecting robots.txt rules for automatic crawls.

- source: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-googlebot-robots
- side: googlebot

> used for Google's products (such as Googlebot). They always respect robots.txt rules for automatic crawls.
### googlebot-issues-requests

Googlebot issues HTTP requests and therefore appears in server access logs under its own user agent.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-googlebot-issues-requests
- side: googlebot
### google-aio-eligibility

Google states that to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet.

- source: https://developers.google.com/search/docs/appearance/ai-features (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-google-aio-eligibility
- side: googlebot
### google-ai-controls

Google lists nosnippet, data-nosnippet, max-snippet and noindex as the controls for limiting a page's use in AI Overviews and AI Mode.

- source: https://developers.google.com/search/docs/appearance/ai-features (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-google-ai-controls
- side: context
### gsc-genai-control

Google's Search Console "Search generative AI" control, rolled out to all websites by 31 August 2026, lets a site owner exclude the site from appearing in and grounding generative AI Search features such as AI Overviews, AI Mode and AI Overviews in Discover; Google says exclusion generally takes a few days.

- source: https://support.google.com/webmasters/answer/16908024 (support.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-gsc-genai-control
- side: context
### study-gx-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 992 (12.3%) disallow Google-Extended at the site root.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-study-gx-blocked
- side: google-extended
### study-gbot-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 156 (1.9%) disallow Googlebot at the site root.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-study-gbot-blocked
- side: googlebot
### dog-googlebot

In the attensira.com workspace, read over the Attensira MCP for the 30 days to 4 October 2026, the AI traffic log recorded 189 Googlebot requests across 80 pages. User agents are self-reported and none was IP-verified.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-dog-googlebot
- side: googlebot
### dog-gx-none

In the attensira.com workspace, read over the Attensira MCP for the 30 days to 4 October 2026, the AI traffic log recorded 0 requests from a separate Gemini or AI Overviews agent among 143,601 AI-crawler requests, consistent with Google-Extended having no user agent of its own.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/google-extended-vs-googlebot#claim-dog-gx-none
- side: google-extended