Bots
Google-Extended: should you block it?
Google-Extended is a robots.txt token, not a crawler: Google uses it to let publishers decide whether crawled content may train future Gemini models. Block it if you object to that training use. Google states it does not affect inclusion in Google Search or ranking, and because it sends no requests, it never appears in your logs.
7 days free · card required
NoneGoogle-Extended sends no requests; crawlers read it from robots.txt
of 143,601 AI crawler requests, none can carry it
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| Google-Extended (this page) | 0 | Allowed |
| meta-externalagent | 131,105 | Allowed |
| GPTBot | ≈830 | Allowed |
| ClaudeBot | ≈68 | Allowed |
| Amazonbot | 0 | Allowed |
| Applebot-Extended | 0 | Allowed |
| MistralAI-Training | 0 | Wildcard |
3 of 7 had requests filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Google-Extended in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Google-Extended today.
# Google-Extended sends no requests. This line changes how# already-crawled pages may be used, and does not affect# Google Search.User-agent: Google-ExtendedDisallow: / # Search indexing is a separate token and a separate# decision.User-agent: GooglebotAllow: /User-agent: Google-ExtendedAllow: /Disallow: /api/We name Google-Extended and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs Google-Extended, and what is it for?
Google runs it, and we file it under training. A control token, not a crawler: it fetches nothing. It tells Google whether content its crawlers already fetched may train future Gemini models, and Google states it does not affect inclusion or ranking in Google Search.
From Google’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
What does a Google-Extended rule actually change?
It changes what Google may do with pages it has crawled, not whether it crawls them. Google documents Google-Extended as a standalone product token publishers use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models2. The crawling itself is done by Googlebot, the agent Google documents as building its search indexes4, and by Google's other agents.
Google also states the token does not affect a site's inclusion in Google Search and is not used as a ranking signal2. Very few operators separate "you may index me" from "you may not train on me" this cleanly, which is why it is worth using instead of a blunter rule. The robots.txt block above disallows Google-Extended and keeps Googlebot allowed.
Getting this wrong produces two opposite errors, both common. Some sites block Googlebot believing it stops training, and lose Search. Others watch their logs for a Google-Extended visit that can never arrive.
Why will Google-Extended never appear in your logs?
Because it makes no requests. Google states that Google-Extended does not have a separate HTTP request user agent string3. Three things follow, and each contradicts something people routinely believe.
- An empty log is the expected result. There is no positive signal to wait for after you add the rule.
- A request claiming to be Google-Extended is fake. Since the real token sends nothing, the header alone proves a spoof. That is rare among crawlers.
- Verification does not apply. Reverse DNS and IP-range checks separate a real fetch from a spoofed one. With no fetch, there is nothing to check. The right check is your own robots.txt: fetch the file and confirm the group parses the way you meant.
Our own logs show the shape. On attensira.com, Attensira's AI traffic measurement over the 30 days to 4 October 2026 recorded 189 Googlebot requests and nothing under a separate Gemini or AI Overviews agent. Google's AI products read the web through the agents that already crawl it.
How many sites use the Google-Extended opt-out?
More than block any search crawler. The study figures below come from our own robots.txt run across the top 10,000 Tranco domains on 3 September 2026: 992 of the 8,036 readable files blocked Google-Extended at the root, against 156 that blocked Googlebot. The common configuration is the intended one: stay in Search, decline training.
Note what "never mentioned" means in those figures. A site that has not written the token has made no decision. It is not consent; it is a policy nobody wrote.
How often do sites block Google-Extended?
Measured by us, not reported by Google: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow Google-Extended at the root
- 12.3%
- 992 of 8,036 domains
- allow it by name
- 3.4%
- 276 domains
- never mention it
- 84.2%
- 6,768 domains
What user-agent string does Google-Extended send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
Is blocking Google-Extended enough to stay out of AI training?
No. It binds Google and nobody else, and there is no shared token for the whole field. Applebot-Extended is the closest analogue: same no-fetch design, different operator, different models. The comparison is in Applebot-Extended vs Google-Extended.
Other operators handle training with an agent that does crawl, such as GPTBot or ClaudeBot, so a rule naming them also changes what appears in your logs. Opting out of training is a list of rules, one per operator, and it says nothing about the training a crawler already did.
One structural warning. Google's crawlers follow the single group with the most specific matching user agent and do not combine it with the wildcard1. A Google-Extended group is therefore read on its own: whatever you put under User-agent: * does not carry over to it. Write the full rule you mean inside each named group.
What should you measure instead of Google-Extended visits?
Two things. First, whether Google still crawls you, and with which agents: Googlebot for Search, GoogleOther for Google's other work. Those do appear in logs and can be verified. Second, whether Google's AI answers name you, which no crawl count can tell you.
Attensira covers both. AI traffic, on Growth and Business, counts the fetches by operator and declared purpose, measured at the edge and reported as the user agents sent. Every plan reads Google AI Overviews daily and shows whether your pages are cited; Gemini is read on Growth and above. See crawler logs, and use the robots.txt generator to write a file that keeps the two groups distinct.
ResearcherTranscribed from Google documentation · Agents filed beside Google-Extended
Should I allow Google-Extended in my situation?
Concrete cases, and what each one implies for this agent.
You added a Google-Extended block a month ago and it has never appeared in your crawler logs, so you assume Google is ignoring it.
Change nothing. Check that the robots.txt group parses instead.
Google documents that Google-Extended has no separate HTTP request user agent string. It never fetches, so it can never be logged. The only thing worth verifying is that your robots.txt actually serves the group you wrote. A misspelt token or a group missing the rule you meant is a real failure, and an empty log is not.
Legal wants your content out of AI training, and someone has proposed disallowing Googlebot to achieve it.
Disallow Google-Extended. Leave Googlebot allowed.
Google documents Google-Extended as the control over whether crawled content may be used for training future generations of Gemini models, and states that it does not affect inclusion in Google Search or act as a ranking signal. Disallowing Googlebot would deindex the site from Search while leaving the Gemini question unanswered.
You disallowed Google-Extended and now want to know whether you are out of AI training generally.
Treat it as one entry on a list, and write the rest of the list.
An opt-out token binds one operator. Google-Extended covers Gemini training and grounding; it says nothing about Apple, OpenAI, Anthropic, Common Crawl or anyone else, each of whom publishes a different token. There is no shared token that covers them all.
What else do people ask?
Does blocking Google-Extended remove me from Google Search or AI Overviews?
Why don't I see Google-Extended in my server logs?
Does blocking Google-Extended opt me out of AI training everywhere?
How many sites block Google-Extended?
Where do these facts come from?
4 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Google documents that only one robots.txt group is valid for a particular crawler. Its crawlers follow the group with the most specific user agent that matches and ignore the others, and user-agent-specific groups are not combined with the * group.
developers.google.com · retrieved
“Only one group is valid for a particular crawler.”
Google documents Google-Extended as a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models, and states that it does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
developers.google.com · retrieved
“Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models.”
Google documents that Google-Extended does not have a separate HTTP request user agent string.
developers.google.com · retrieved
“Doesn't have a separate HTTP request user agent string”
Google documents Googlebot as the crawler that finds information for building Google's search indexes.
developers.google.com · retrieved
“Find information for building Google's search indexes”
Want whatever it takes to win the answer?
7 days free · card required