Bots
Applebot-Extended: should you block it?
Applebot-Extended is a robots.txt token, not a crawler: Apple uses it to decide whether data Applebot crawled may help train Apple's foundation models. Block it if you object to that use. Apple states it does not crawl webpages and that pages disallowing it can still appear in search results, so your Siri, Spotlight and Safari presence stays.
7 days free · card required
NoneApplebot-Extended sends no requests; crawlers read it from robots.txt
of 143,601 AI crawler requests, none can carry it
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| Applebot-Extended (this page) | 0 | Allowed |
| meta-externalagent | 131,105 | Allowed |
| GPTBot | ≈830 | Allowed |
| ClaudeBot | ≈68 | Allowed |
| Amazonbot | 0 | Allowed |
| Google-Extended | 0 | Allowed |
| MistralAI-Training | 0 | Wildcard |
3 of 7 had requests filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Applebot-Extended in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Applebot-Extended today.
# Apple's foundation-model opt-out. It sends no requests,# and Apple states that pages disallowing it can still# appear in search results.User-agent: Applebot-ExtendedDisallow: / # Apple's actual crawler, and the reason you are in# Spotlight, Siri and Safari.User-agent: ApplebotAllow: /User-agent: Applebot-ExtendedAllow: /Disallow: /api/We name Applebot-Extended and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs Applebot-Extended, and what is it for?
Apple runs it, and we file it under training. A control token, not a crawler: it fetches nothing. It tells Apple whether pages Applebot already crawled may help train Apple's foundation models, and Apple states a page that disallows it can still appear in search results.
From Apple’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Why is Applebot-Extended a rule about use, not about crawling?
Because Apple crawls once and decides use afterwards. Apple says the data Applebot crawls may be used both for search in Spotlight, Siri and Safari and to help train Apple's foundation models. Applebot-Extended is how you separate those two uses. Apple documents it as the way publishers opt out of having content crawled by Applebot used to help train its foundation models, and says it is only used to determine how to use the data crawled by the Applebot user agent1.
Apple puts the mechanism in four words: Applebot-Extended does not crawl webpages2. It never fetches anything. Your robots.txt group is read by Applebot and applied to what it collects.
That is the opposite of the training crawlers some other operators run, such as GPTBot or ClaudeBot, where the training opt-out stops a crawler that would otherwise visit. With Apple, blocking Applebot-Extended does not reduce crawling at all. Applebot still comes, because you still want it for search.
Does blocking it take you out of Apple search?
No. Apple states that webpages which disallow Applebot-Extended can still be included in search results3. The indexing that feeds Spotlight, Siri and Safari is Applebot's.
This is the separation people reach for and miss. Given the brief "keep us out of Apple Intelligence", the instinct is to block Applebot. That achieves the goal and deletes you from Apple's search surfaces. The robots.txt block above does it the right way: Applebot-Extended disallowed, Applebot allowed. Apple documents the group as the way to express the opt-out4.
Check one more thing. Applebot follows your Googlebot rules when you have no Applebot group, so an explicit Applebot group keeps an unrelated Googlebot edit from moving Apple's crawler.
Why will Applebot-Extended never show in your logs?
Because it is not a fetcher. Three consequences follow:
- An empty log is the correct outcome. There is no positive signal to wait for after you add the rule.
- A log line claiming to be Applebot-Extended is fake. The real token sends nothing, so the header alone settles it.
- Verification does not apply. The reverse DNS and CIDR checks separate a real fetch from a spoofed one. With no fetch, there is nothing to separate. The check that does exist is your own file: fetch your robots.txt and confirm the Applebot-Extended group parses as its own group.
What user-agent string does Applebot-Extended send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
Can you opt out for only part of your site?
Yes, and it is often the better answer. Apple's own example scopes the rule to a path, Disallow: /private/ under User-agent: Applebot-Extended, which shows the opt-out takes ordinary path rules. You can withhold a subscriber archive or a paid research section from training and leave public marketing pages available.
One structural warning: a named group replaces the wildcard for that agent; it does not add to it. Under the robots.txt standard, a crawler follows the User-agent: * group only when no group names it. So a User-agent: * Disallow: / elsewhere in your file applies to Applebot only if Applebot has no group of its own, and once you write an Applebot-Extended group, it carries every rule you want for that token. Read the finished file back.
How does Applebot-Extended compare with Google-Extended?
They have the same shape, a robots.txt token with no crawler behind it, and cover different operators and different models. Google documents Google-Extended as its own standalone token for whether crawled content may be used to train future Gemini models5. Neither says anything about the other, and there is no shared token for the field. The pair is compared in Applebot-Extended vs Google-Extended, and Google's side has its own page at Google-Extended.
Our robots.txt study did not include Applebot-Extended in its 3 September 2026 run, so there is no adoption figure here, and we will not estimate one. For Google-Extended, the same run found 992 of 8,036 readable files among the top 10,000 Tranco domains blocking it.
What should you measure instead?
Whether Apple still crawls you, and how much. That is a question about Applebot, which does appear in logs and can be verified against Apple's CIDR file. AI traffic in Attensira, on Growth and Business, counts crawler fetches per operator and declared purpose at the edge, so you can watch the crawling that continues after a training opt-out instead of waiting for a token that never arrives. It reports user agents as sent. See crawler logs and the robots.txt generator.
ResearcherTranscribed from Apple documentation · Agents filed beside Applebot-Extended
In this entry
Should I allow Applebot-Extended in my situation?
Concrete cases, and what each one implies for this agent.
You disallowed Applebot-Extended weeks ago and it has never appeared in your logs, so you suspect Apple is ignoring it.
Change nothing about the rule. Verify that your robots.txt serves the group instead.
Apple states that Applebot-Extended does not crawl webpages, so it can never produce a log line. The absence is what compliance looks like. What can genuinely fail is the file — a merged group, a stray character, a CDN serving a cached older robots.txt — and that is checkable in a way the logs are not.
Someone proposes disallowing Applebot to keep your content out of Apple Intelligence.
Disallow Applebot-Extended instead, and leave Applebot allowed.
Apple documents Applebot as the crawler powering Spotlight, Siri and Safari search, and Applebot-Extended as the separate control over training use which does not remove pages from search results. Blocking Applebot achieves the training goal by also deleting yourself from Apple's search surfaces, which is a much larger bill than anyone intended.
You have written an Applebot-Extended rule and want to know whether you are out of AI training generally.
Treat it as one line item and enumerate the rest, operator by operator.
An opt-out token binds one company. Applebot-Extended covers Apple's foundation models and says nothing about Google's Gemini, which has its own Google-Extended token, or about any other operator. There is no shared token that covers the field, so the opt-out is a list you maintain rather than a switch you flip.
What else do people ask?
Does blocking Applebot-Extended remove me from Siri or Spotlight?
Why don't I see Applebot-Extended in my logs?
Can I opt out of Apple training for part of my site?
How many sites block Applebot-Extended?
Where do these facts come from?
5 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Apple documents Applebot-Extended as the way publishers opt out of having content crawled by Applebot used to help train Apple's foundation models, and states that it is only used to determine how to use the data crawled by the Applebot user agent.
support.apple.com · retrieved
“Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.”
Apple states that Applebot-Extended does not crawl webpages.
support.apple.com · retrieved
“Applebot-Extended does not crawl webpages.”
Apple states that webpages which disallow Applebot-Extended can still be included in search results, and documents Applebot as the crawler whose data powers the search technology in Spotlight, Siri and Safari.
support.apple.com · retrieved
“Webpages that disallow Applebot-Extended can still be included in search results.”
Apple states that Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot, and documents an Applebot-Extended robots.txt group as the way to express the training opt-out.
support.apple.com · retrieved
“Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot.”
Google documents Google-Extended as a separate standalone product token covering whether crawled content may be used for training future generations of Gemini models, which is a different operator and a different set of models from Apple's.
developers.google.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required