Bots
AI training crawlers
AI training bots collect content that may be used to train models. GPTBot, ClaudeBot, Amazonbot, MistralAI-Training and meta-externalagent crawl; Google-Extended and Applebot-Extended are opt-out tokens that send no requests. Because each operator gives search its own token, you can refuse training and stay citable. Read meta-externalagent first: Meta says it also indexes content.
7 days free · card required
- agents documented
- 7
- publish IP ranges or reverse DNS
- 3 of 7
- requests to attensira.com, 30 days
- ≈132,000
- of n=143,601 AI crawler requests
- of top sites disallow GPTBot, the most blocked of the 4 in the study
- 14.3%
- n=8,036 robots.txt files
One of the category's rules, from the GPTBot entry, per the operator's documentation; the other 6 are on their own entries.
User-agent: GPTBotDisallow: /
ResearcherOperator documentation, our AI traffic board (Attensira MCP) and robots.txt study · Which AI training crawlers and tokens are documented?
Collects text that may enter a training corpus. Blocking it removes no page from any assistant's search index or citations, though a model can still name you from what it already learned.
- Applebot-Extended Applebot-Extended is Apple's robots.txt opt-out from foundation model training. It never crawls, never shows in logs, and keeps you in Apple search. Apple · Training
- Google-Extended Google-Extended is the robots.txt token that keeps your pages out of Gemini training. It sends no requests and does not affect Google Search. Google · Training
- meta-externalagent Meta documents meta-externalagent as crawling for AI training or direct indexing. The robots.txt block, why it cannot be verified, and what we see on our site. Meta · Training
- MistralAI-Training MistralAI-Training collects data to train Mistral's models. Blocking it leaves Mistral search and live answers intact. The rule to copy. Mistral AI · Training
- Amazonbot Amazonbot improves Amazon products and may train Amazon AI models. How to block it but stay in Alexa search, the 30-day robots.txt cache, and verification. Amazon · Training
- ClaudeBot ClaudeBot is Anthropic's training crawler. What it collects, the robots.txt rule, Crawl-delay, and why blocking it keeps you findable in Claude. Anthropic · Training
- GPTBot GPTBot is OpenAI's training crawler. What it collects, the robots.txt rule, how to check a visit, and why blocking it leaves you in ChatGPT. OpenAI · Training
Which AI training crawlers fetched attensira.com in the last 30 days?
3 of 7 had requests filed on attensira.com in the window. User agents as sent, not checked against address lists.
| Agent | Requests | Our robots.txt |
|---|---|---|
| meta-externalagent | 131,105 | Allowed by name |
| GPTBot | ≈830 | Allowed by name |
| ClaudeBot | ≈68 | Allowed by name |
| Applebot-Extended | 0 | Allowed by name |
| Google-Extended | 0 | Allowed by name |
| MistralAI-Training | 0 | No named group |
| Amazonbot | 0 | Allowed by name |
How do the AI training crawlers compare?
Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.
Sorted by top sites blocking, high to low.
| Agent | |||||
|---|---|---|---|---|---|
| GPTBot | OpenAI | Respected | IP ranges | Allowed by name | 14.3% |
| ClaudeBot | Anthropic | Respected | IP ranges | Allowed by name | 13.4% |
| meta-externalagent | Meta | Respected | Nothing published | Allowed by name | 12.4% |
| Google-Extended | Respected | Nothing published | Allowed by name | 12.3% | |
| Applebot-Extended | Apple | Respected | Nothing published | Allowed by name | Not in study |
| MistralAI-Training | Mistral AI | Respected | Nothing published | No named group | Not in study |
| Amazonbot | Amazon | Respected | IP ranges | Allowed by name | Not in study |
GPTBot
- Operator
- OpenAI
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Allowed by name
- Top sites blocking
- 14.3%
ClaudeBot
- Operator
- Anthropic
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Allowed by name
- Top sites blocking
- 13.4%
meta-externalagent
- Operator
- Meta
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- Allowed by name
- Top sites blocking
- 12.4%
Google-Extended
- Operator
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- Allowed by name
- Top sites blocking
- 12.3%
Applebot-Extended
- Operator
- Apple
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- Allowed by name
- Top sites blocking
- Not in study
MistralAI-Training
- Operator
- Mistral AI
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- No named group
- Top sites blocking
- Not in study
Amazonbot
- Operator
- Amazon
- robots.txt
- Respected
- Verify by
- IP ranges
- attensira.com
- Allowed by name
- Top sites blocking
- Not in study
Each row read from the operator's documentation, 3 Sep 2026.
“Not in study” means the token is not one our robots.txt study counts yet, not that no site blocks it.
Does blocking a training crawler hurt AI visibility?
Not when the operator gives search its own token. A training fetch takes text into a corpus and returns no link to a reader. OpenAI's GPTBot, Anthropic's ClaudeBot, Amazon's Amazonbot and Mistral's MistralAI-Training each have a separate search or user-triggered token beside them, so a Disallow naming only the training token leaves the citation path open.
meta-externalagent is the exception to read first. Meta documents it as crawling for use cases such as training foundation AI models or indexing content directly. Meta AI's search crawler is meta-webindexer, a separate token.
Meta publishes no reverse-DNS suffix and no IP range file for meta-externalagent, so a visit can be matched only by its user agent.
The meta-externalagent entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Meta documentation · Why will I never see Google-Extended in my logs?
Because it sends no requests. Google documents Google-Extended as a robots.txt token that manages whether content Google already crawls may be used to train future Gemini models, and says it does not affect Google Search. Apple says the same shape of thing about Applebot-Extended: it does not crawl, and pages that disallow it can still appear in Apple's search results. A log line claiming either name is something else.
Google publishes no reverse-DNS suffix and no IP range file for Google-Extended, so a visit can be matched only by its user agent.
The correct check is not in your logs but in your robots.txt. Fetch your own file and confirm the group parses as you intended. The Google-Extended entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Google documentation · What does attensira.com do with training crawlers?
It allows six of them by name, except /api/; MistralAI-Training has no group and falls to the wildcard, which allows it on the same terms. The matrix above reads both from the live robots.txt. The one training source it blocks site-wide is CCBot, the Common Crawl archive that many training datasets are built from. That is a choice about contributing to shared models, not a visibility tactic, and it can be reversed for any one operator without touching that operator's search token.
Mistral AI publishes no reverse-DNS suffix and no IP range file for MistralAI-Training, so a visit can be matched only by its user agent.
The MistralAI-Training entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Mistral AI documentation · How often do the most visited sites block AI training crawlers?
| Name | Disallowed at the root | Count |
|---|---|---|
| GPTBot | 14.3% | 1,147 |
| ClaudeBot | 13.4% | 1,077 |
| meta-externalagent | 12.4% | 994 |
| Google-Extended | 12.3% | 992 |
Web engineer · Writes robots rules like theseRobots.txt access study, our own fetch · · n=8,036 robots.txt filesGPTBot is the most blocked here: 1,147 of 8,036 sites disallow it at the root. The least blocked, Google-Extended, is disallowed by 992.
A site that never names an agent is counted as unspecified, not as allowing it.
What else do people ask?
What are AI training bots?
Does blocking GPTBot remove me from ChatGPT?
Should I block AI training crawlers?
Why can't I verify meta-externalagent or MistralAI-Training?
Where do these facts come from?
9 claims on this page, each with the page it was read from; last checked .
Apple documents Applebot-Extended as the way publishers opt out of having content crawled by Applebot used to help train Apple's foundation models, and states that it is only used to determine how to use the data crawled by the Applebot user agent.
support.apple.com · retrieved
“Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.”
Google documents Google-Extended as a separate standalone product token covering whether crawled content may be used for training future generations of Gemini models, which is a different operator and a different set of models from Apple's.
developers.google.com · retrieved
Google documents that only one robots.txt group is valid for a particular crawler. Its crawlers follow the group with the most specific user agent that matches and ignore the others, and user-agent-specific groups are not combined with the * group.
developers.google.com · retrieved
“Only one group is valid for a particular crawler.”
Meta states that the Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.
developers.facebook.com · retrieved
“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
OpenAI documents GPTBot as the agent used to crawl content that may be used in training its generative AI foundation models.
developers.openai.com · retrieved
“crawl content that may be used in training our generative AI foundation models”
Mistral states that MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models.
docs.mistral.ai · retrieved
“MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models.”
Amazon states that Amazonbot is used to improve its products and services, helping provide more accurate information to customers, and may be used to train Amazon AI models.
developer.amazon.com · retrieved
“Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models.”
Anthropic documents ClaudeBot as helping enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training.
support.claude.com · retrieved
“helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training”
RFC 9309 section 2.2.1 states that a crawler obeys the group with a * user agent only when no group matches its own product token. Its rule for combining groups covers several groups that match the same token.
rfc-editor.org · retrieved
“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”
Want whatever it takes to win the answer?
7 days free · card required