# Applebot-Extended vs Google-Extended: two opt-outs that crawl nothing

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> Two AI opt-out tokens that crawl nothing. What Applebot-Extended and Google-Extended each control, what they leave alone, and how to write both.

- Canonical page: https://attensira.com/compare/applebot-extended-vs-google-extended
- Family: compare
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=applebot-extended-vs-google-extended

**In short:** Applebot-Extended and Google-Extended are robots.txt tokens, not crawlers. Each tells its company whether pages its main crawler already fetched may be used by its AI models. Apple's covers training of its foundation models, including Apple Intelligence. Google's covers Gemini training and grounding. Blocking either leaves you in Apple's and Google's search results.

**Readings, 3 Sep 2026**

| Measure | Value |
| --- | --- |
| of top-10k domains disallow Google-Extended at the root (992) (n=8,036 domains) | 12.3% |
| allow Google-Extended by name (276) (n=8,036 domains) | 3.4% |
| Applebot-Extended; the study's token list did not include it | Not read |

_Attensira robots.txt study, run of 3 September 2026 · 3 Sep 2026 · n=8,036_

## Verdict

Two companies solved the same problem the same way: a token with no crawler behind it, read by the main crawler, governing AI use of what was fetched. The scopes differ. Apple's covers training of its foundation models. Google's covers Gemini training and grounding in Gemini Apps and Vertex AI.

## The readings on this page

### The readings at the top

robots.txt of the top-10,000 Tranco domains (list K9QPW), read by Attensira · 3 Sep 2026 to 3 Sep 2026

| Reading | Value | Sample |
| --- | --- | --- |
| of top-10k domains disallow Google-Extended at the root (992) | 12.3% | n=8,036 domains |
| allow Google-Extended by name (276) | 3.4% | n=8,036 domains |
| Applebot-Extended; the study's token list did not include it | Not read |  |

### Do these tokens ever show up in a log?

attensira.com AI traffic log, read over the Attensira MCP; neither token sends requests of its own · 5 Sep 2026 to 4 Oct 2026

| Reading | Value | Sample |
| --- | --- | --- |
| requests labelled Applebot-Extended on attensira.com | 0 | n=143,601 AI-crawler requests |
| requests labelled Google-Extended on attensira.com | 0 | n=143,601 AI-crawler requests |

## How do Applebot-Extended and Google-Extended compare?

Each cell cites a claim in the Sources section below.

| Criterion | Applebot-Extended | Google-Extended |
| --- | --- | --- |
| **What it governs** |  |  |
| What it governs | Training of Apple's foundation models | Gemini training, Vertex AI training, and grounding |
| Covers grounding at answer time as well as training | Not described in Apple's documentation | Yes · Grounding is named explicitly |
| Products named in the opt-out's scope | Apple Intelligence, Services, Developer Tools | Gemini Apps, Vertex AI API for Gemini |
| Operator states search inclusion is unaffected | Yes · Pages can still be in search results | Yes · Not inclusion, not a ranking signal |
| **How the opt-out works** |  |  |
| Does it fetch pages? | No · Apple states it does not crawl | No · No user-agent string of its own |
| The crawler that actually fetches | Applebot, which powers Spotlight, Siri and Safari | Googlebot, which powers Search, Discover, Images, Video and News |
| How the opt-out is written | robots.txt group for Applebot-Extended | robots.txt group for Google-Extended |
| **What we measured** |  |  |
| Top-10k domains disallowing it at the root (study, n=8,036) | Not measured in this run | 992 (12.3%) |
| Requests labelled with the token on attensira.com, 30 days | 0 | 0 |

## What is Applebot-Extended?

Apple's secondary user agent. It crawls nothing and exists to let publishers say whether content already fetched by Applebot may train Apple's foundation models for generative features.

## What is Google-Extended?

Google's standalone product token. It has no user-agent string of its own and governs whether crawled content may train Gemini models or be used for grounding in Gemini Apps and Vertex AI.

## When should you choose Applebot-Extended?

Disallow Applebot-Extended to keep your pages out of Apple's foundation-model training. That covers the models behind Apple Intelligence, and Apple says pages that disallow it can still appear in search results in Spotlight, Siri and Safari.

## When should you choose Google-Extended?

Disallow Google-Extended to keep them out of Gemini training and grounding. That covers Gemini Apps and Vertex AI answers, and Google says it does not affect inclusion in Google Search and is not a ranking signal.

## When is neither the right choice?

Neither token stops Applebot or Googlebot from fetching your pages, and neither removes you from search or from Google's AI Overviews. To keep a page out of Google's search-side AI features, the controls are noindex and the snippet rules, set in the page.

## Where does Attensira fit?

Worth a look if you want crawl decisions judged on answers: Attensira counts crawler requests at your edge and reads the engines it tracks daily. Whatever it takes to win the answer.

## What sets Applebot-Extended and Google-Extended apart?

- Neither token can be watched working: no request ever carries either name, so the only evidence an opt-out is honoured is the operator's word and, over time, what its assistants say.
- Lists of AI crawlers to block often name Google-Extended and leave Apple out, so many sites have opted out of Gemini use while staying in Apple's training without anyone deciding it.
- The decision is about what you sell, not about search: a docs-led product may want to be learned, a publisher whose writing is the revenue may not.

## Why compare two tokens from two companies?

Because Apple and Google solved the same problem in the same way, and the shared design is the point. Both already run a crawler the web depends on: Applebot powers search features across Apple's ecosystem, including Spotlight, Siri and Safari, and Googlebot powers Google Search. Neither company wanted publishers to choose between search and AI. So each added a token that crawls nothing and only governs how pages already fetched may be used by its models.

Apple calls Applebot-Extended a secondary user agent that gives publishers additional controls over how their content can be used, and says plainly that it does not crawl webpages. Google calls Google-Extended a standalone product token and says it has no separate HTTP user-agent string; crawling is done by Google's existing user agents. The two are mirror images, and understanding one tells you most of what you need to know about the other.

## What does each token actually control?

Applebot-Extended controls whether your content may be used to train Apple's general-purpose foundation models, which power generative features across Apple products, including Apple Intelligence, Services and Developer Tools. Disallow it, and Apple says your pages can still be included in search results.

Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and the Vertex AI API for Gemini. Grounding means supplying content at answer time to make a response more accurate. Google says the token does not affect inclusion in Google Search and is not a ranking signal.

| What the token reaches | Applebot-Extended | Google-Extended |
| --- | --- | --- |
| Model training | Apple's foundation models | Future Gemini models |
| Answer-time grounding | Not described by Apple | Gemini Apps and Vertex AI |
| Search inclusion | Unaffected, per Apple | Unaffected, per Google |
| Crawling | None; Applebot fetches | None; Google's crawlers fetch |

The difference in scope matters. Google's token reaches answer-time use in Gemini products as well as training. Apple's documentation describes its token in terms of training. If your concern is an assistant reading your current pages to answer a question, Google's token speaks to it for Gemini; Apple's documentation does not describe an equivalent.

## Why is a token that crawls nothing a good design?

Because it separates two decisions publishers want to make separately. Before these tokens existed, the only lever was the crawler itself: block Googlebot and lose Search, or allow it and accept every downstream use. A usage token lets the crawler keep fetching for search while the publisher says no to AI use. The opt-out costs nothing in search terms, which is why it spread quickly.

The trade-off is that nothing can be observed. A token that makes no requests never shows up. On attensira.com, 0 of 143,601 AI-crawler requests in the 30 days to 4 October 2026 were labelled Applebot-Extended or Google-Extended. That zero is not a finding about Apple or Google; it is what a token looks like from the server. You rely on the companies' documentation and on their honouring of it.

### Do these tokens ever show up in a log?

No. In the 30 days to 4 October 2026, not one of 143,601 AI-crawler requests to attensira.com was labelled Applebot-Extended or Google-Extended. That zero is what a usage token looks like from the server, not a finding about Apple or Google.

**Readings, 5 Sep 2026 to 4 Oct 2026**

| Measure | Value |
| --- | --- |
| requests labelled Applebot-Extended on attensira.com (n=143,601 AI-crawler requests) | 0 |
| requests labelled Google-Extended on attensira.com (n=143,601 AI-crawler requests) | 0 |

_AI-crawler requests to attensira.com, Attensira MCP, attensira.com workspace · 4 Oct 2026 · n=143,601_

## What should your robots.txt say?

For a site that wants to stay in Apple's and Google's search results but keep its pages out of both companies' AI use:

```
# AI use only. These tokens do not affect search.
User-agent: Applebot-Extended
Disallow: /

User-agent: Google-Extended
Disallow: /
```

Do not add Applebot or Googlebot to those groups; they are the crawlers behind search. To keep only a section out, scope each group to paths instead of the root. Both files are per host: a blog on a subdomain needs the same lines in its own robots.txt.

Comment each group with the question it answers, who decided and when. A future editor who sees a "block AI" comment above Googlebot will either break your search visibility or delete a deliberate rule, and the comment is the cheapest protection against both.

## How do large sites use these opt-outs?

On 3 September 2026 we read the robots.txt of the top 10,000 domains in a dated Tranco list and could read 8,036 of them. 992 (12.3%) disallow Google-Extended at the root, close to the rates for the labs' own training crawlers (14.3% for GPTBot, 13.4% for ClaudeBot) and far above the 1.9% that disallow Googlebot. Most still never name Google-Extended at all: 84% leave it to their wildcard group. Large sites that do decide treat Google-Extended as a training decision and Googlebot as a search decision, which is exactly what Google intends.

Applebot-Extended was not among the 23 tokens in that run, so we publish no rate for it and will not estimate one. If you want to know whether a peer set uses it, the method on our study page can be repeated against any list of domains.

## Why is Apple's token the one people forget?

Because Google's is better known, and Apple's arrives through a crawler most site owners never think about. Applebot is rarely the bot a team looks for in its logs, and Apple's AI features live inside devices rather than on a website people visit. Lists of "AI crawlers to block" often name Google-Extended and leave Apple out, or list Applebot-Extended as if it were a crawler you could watch for in a log.

The result is asymmetric files written by accident: a site opts out of Gemini training and stays in Apple's without anyone deciding to. If the two companies should follow the same policy on your site, write both groups on the same day and in the same block. Then read the live file from the public host, not from your repository, because a CDN or a framework default can serve something other than what you committed.

## What do these tokens not do?

| A common belief | What the documentation says |
| --- | --- |
| Disallowing them stops the crawler | No; Applebot and Googlebot keep fetching |
| They remove content already used | No; both are forward-looking |
| They cost search visibility | No, by both companies' statements |
| One company's token covers the other | No; each reads only its own |

They also do not touch Google's search-side AI features. AI Overviews and AI Mode are Search features. Google says a page must be indexed and eligible to be shown with a snippet to appear as a supporting link in them, and names noindex, nosnippet, max-snippet and data-nosnippet as the ways to limit that. If your goal is to stay out of AI Overviews, Google-Extended is the wrong tool.

## What can you measure, and what can you not?

You can measure the crawlers. Googlebot shows up under its own name: on attensira.com it made 189 requests across 80 pages in the 30 days to 4 October 2026. Applebot is harder; the AI traffic board we read classifies AI operators, and Applebot is not one of the agents it reported for our site in that window, so we have no Applebot figure to give you. If Apple matters to your buyers, check your raw server logs for its user agent and verify it the way Apple documents.

You cannot measure whether a company honours a usage token. There is no request to see and no output to inspect for a single site. What you can do is read answers. Ask the assistants your buyers use the questions they ask, on a schedule, and keep the answers. Our own workspace reads ChatGPT and Perplexity daily, so we can say what those two engines cite from attensira.com; we do not read Apple Intelligence or Gemini, and we will not publish a number for either.

That gap is worth stating plainly because it shapes the decision. A training opt-out is a statement of policy whose effect you will never see directly. Make it for the reason you hold, write the reason down, and judge your visibility by what answers say, not by what your file says.

## Who should allow both tokens?

Companies that would rather be learned than protected, and the reverse for companies whose writing is the product. Nobody can measure the effect of either choice for a single site, and neither company claims one, so the decision rests on what you sell:

| Kind of site | Usual choice for both tokens | Why |
| --- | --- | --- |
| Developer tool whose docs are the product | Allow | Being learned is distribution |
| Reference publisher that wants its definitions repeated | Allow | Same reason |
| Local business described by assistants on phones | Allow | Accurate descriptions help |
| Publisher whose writing is its revenue | Disallow | Answers may replace the visit |
| Licensed or paywalled content | Disallow, plus authentication | The token is not access control |

A publisher that disallows both should still keep the search crawlers allowed, so the visit can happen when an answer links to it. For a typical B2B company the decision is usually: allow both search crawlers, decide the two tokens together and in writing, then put the effort into the pages. Visibility in AI answers is decided mostly by search-side retrieval and by third-party pages that describe your category, not by whether one company's foundation model trained on your pricing page.

## Is there a reason to allow one and block the other?

Sometimes. A company whose buyers use Gemini to research suppliers might allow Google-Extended so Gemini app answers can be grounded on its current pages, while disallowing Applebot-Extended because its concern with Apple is training only. A company with a licensing position on one of the two might do the reverse. Those are specific, legitimate reasons. Write them next to the rule, and revisit both tokens whenever either company changes the scope it describes, since a scope that grows can turn yesterday's reasonable allow into something you would not have agreed to.

## FAQ

### Do Applebot-Extended and Google-Extended crawl my site?

No. Apple states that Applebot-Extended does not crawl webpages, and Google states that Google-Extended has no separate user-agent string; crawling is done by Google's existing user agents. Both are tokens the main crawler's owner reads when deciding how pages it already fetched may be used.

### Will blocking them remove me from Siri, Spotlight or Google Search?

No, by both companies' documentation. Apple says webpages that disallow Applebot-Extended can still be included in search results. Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.

### What exactly does each token control?

Applebot-Extended controls whether your content may be used to train Apple's general-purpose foundation models powering generative features across Apple products, including Apple Intelligence, Services and Developer Tools. Google-Extended controls Gemini model training and grounding in Gemini Apps and the Vertex AI API for Gemini.

### Does one token cover the other company?

No. Each company reads only its own token. A Google-Extended rule says nothing to Apple, and an Applebot-Extended rule says nothing to Google. If you want both companies to follow the same policy, write both groups.

### Will I see Applebot-Extended or Google-Extended in my server logs?

No. Neither makes requests. On attensira.com, 0 of 143,601 AI-crawler requests in the 30 days to 4 October 2026 were labelled with either token. You see Applebot and Googlebot, the crawlers that fetch, so logs cannot show whether an opt-out is honoured.

### Does Google-Extended keep my pages out of AI Overviews?

No. AI Overviews and AI Mode are Google Search features, and Google says a page must be indexed and eligible for a snippet to appear in them. The controls for those are noindex, nosnippet, max-snippet and data-nosnippet, read by Googlebot, not Google-Extended.

## Sources

Each entry below is one claim on this page. `retrieved` is the date the exporter or a human last saw the source say this, and nothing more: it is not a publication date and not a promise of re-verification. `cite` is the anchor on the canonical page where this claim is rendered.

### ax-not-crawler

Apple states that Applebot-Extended does not crawl webpages and is a secondary user agent giving publishers additional controls over how content can be used.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-not-crawler
- side: applebot-extended

> Applebot-Extended does not crawl webpages.
### ax-controls

Apple documents Applebot-Extended as determining whether content trains Apple's foundation models for generative features.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-controls
- side: applebot-extended
### ax-search-safe

Apple states that webpages which disallow Applebot-Extended can still be included in search results.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-search-safe
- side: applebot-extended

> Webpages that disallow Applebot-Extended can still be included in search results.
### ax-directive

Apple publishes the opt-out directive as a robots.txt block using the user-agent token Applebot-Extended with Disallow set to the whole site.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-directive
- side: applebot-extended
### ax-parent

Apple documents Applebot, the parent crawler, as powering search technology across Apple's ecosystem including Spotlight, Siri and Safari.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-parent
- side: applebot-extended

> the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari
### ax-grounding-unstated

Apple's Applebot documentation describes Applebot-Extended in terms of training foundation models and does not describe a grounding or retrieval use.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-grounding-unstated
- side: applebot-extended
### ax-scope-apple-intelligence

Apple states that disallowing Applebot-Extended prevents content being used to train Apple's general-purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services and Developer Tools.

- source: https://support.apple.com/en-us/119829 (support.apple.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-ax-scope-apple-intelligence
- side: applebot-extended
### gx-not-crawler

Google states that Google-Extended does not have a separate HTTP request user-agent string, and that crawling is conducted using existing Google user agents.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-gx-not-crawler
- side: google-extended

> Google-Extended doesn't have a separate HTTP request user agent string
### gx-controls

Google documents Google-Extended as a standalone product token for managing whether crawled content may be used to train future generations of Gemini models.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-gx-controls
- side: google-extended

> Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models
### gx-search-safe

Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-gx-search-safe
- side: google-extended

> Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search
### gx-directive

Google-Extended is applied as a robots.txt user-agent token in the ordinary way.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-gx-directive
- side: google-extended
### gx-parent

Google documents Googlebot, the crawler whose fetches Google-Extended governs, as affecting Google Search, Discover, Images, Video and News.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-gx-parent
- side: google-extended
### gx-grounding

Google documents Google-Extended as covering grounding in Gemini Apps and Vertex AI as well as model training.

- source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (developers.google.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-gx-grounding
- side: google-extended
### study-gx-blocked

Of the 8,036 top-10,000 Tranco domains whose robots.txt Attensira read on 3 September 2026, 992 (12.3%) disallow Google-Extended at the site root.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-study-gx-blocked
- side: google-extended
### study-ax-unmeasured

Attensira's September 2026 robots.txt study of 8,036 top-10,000 domains did not include Applebot-Extended among the tokens measured.

- source: https://attensira.com/data/ai-crawler-robots-access (attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-study-ax-unmeasured
- side: applebot-extended
### dog-ax-none

In the attensira.com workspace, read over the Attensira MCP for the 30 days to 4 October 2026, no request among 143,601 AI-crawler requests was labelled Applebot-Extended.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-dog-ax-none
- side: applebot-extended
### dog-gx-none

In the attensira.com workspace, read over the Attensira MCP for the 30 days to 4 October 2026, no request among 143,601 AI-crawler requests was labelled Google-Extended.

- source: https://docs.attensira.com/mcp/tools (docs.attensira.com)
- retrieved: 2026-10-04
- cite: https://attensira.com/compare/applebot-extended-vs-google-extended#claim-dog-gx-none
- side: google-extended