Bots
Search engine crawlers
Search engine crawlers build the classic search indexes: Googlebot and its image and video variants, bingbot, Applebot, Kagibot and PetalBot. Google's generic crawlers, GoogleOther and GoogleOther-Image, are filed here too. Disallowing Googlebot or bingbot stops that engine reading your pages, though a blocked URL can still be listed without a description. Training has separate tokens: Google-Extended and Applebot-Extended.
7 days free · card required
- agents documented
- 9
- publish IP ranges or reverse DNS
- 9 of 9
- requests to attensira.com, 30 days
- 730
- of n=143,601 AI crawler requests
- of top sites disallow bingbot, the most blocked of the 2 in the study
- 2.2%
- n=8,036 robots.txt files
One of the category's rules, from the Googlebot entry, per the operator's documentation; the other 8 are on their own entries.
User-agent: GooglebotAllow: /
ResearcherOperator documentation, our AI traffic board (Attensira MCP) and robots.txt study · Which search engine crawlers are documented?
Search indexing
Builds a search engine's index. Blocking a primary crawler stops that engine, and features built on its index such as Google's AI Overviews, from reading your pages; a media or regional variant scopes only its own results.
- Googlebot-Image Googlebot-Image fetches images for Google Images, Discover and the favicons and logos in Search. What blocking it costs and how to verify it. Google · Search indexing
- Googlebot-Video Googlebot-Video fetches video files for Google's video features. Why it is a bandwidth decision first, what blocking costs, and how to verify it. Google · Search indexing
- Kagibot Kagibot indexes the web for Kagi, a paid search engine. Its header, the four published addresses, and the Googlebot fallback that already governs it. Kagi · Search indexing
- PetalBot PetalBot indexes for Petal Search, which Huawei says also powers Huawei Assistant and AI Search. Token, headers, what blocking costs, and how to verify it. Huawei · Search indexing
- Applebot Applebot feeds Spotlight, Siri and Safari search and may help train Apple's models. Its Googlebot fallback rule, the training opt-out, and verification. Apple · Search indexing
- bingbot bingbot is Microsoft's crawler for the Bing index. Why to allow it, the bingbot.json range file Bing now publishes, and how to verify a visit. Microsoft · Search indexing
- Googlebot Googlebot builds Google's search index, which AI Overviews draw on. Why blocking it is not a Gemini opt-out, and how to verify a real visit. Google · Search indexing
Product operations
Google's generic product-team crawling. Google states that rules addressed to it do not affect any specific product.
- GoogleOther-Image GoogleOther-Image is Google's generic image fetcher, not the Google Images crawler. How to block it without losing image search, and how to check it. Google · Product operations
- GoogleOther GoogleOther is Google's generic crawler for product teams and internal research. Why blocking it does not touch Search, and how to verify it. Google · Product operations
Which search engine crawlers fetched attensira.com in the last 30 days?
2 of 9 had requests filed on attensira.com in the window. User agents as sent, not checked against address lists.
| Agent | Requests | Our robots.txt |
|---|---|---|
| bingbot | 541 | Allowed by name |
| Googlebot | 189 | Allowed by name |
| Googlebot-Image | 0 | Follows Googlebot |
| Googlebot-Video | 0 | Follows Googlebot |
| GoogleOther-Image | 0 | Follows GoogleOther |
| Kagibot | 0 | No named group |
| PetalBot | 0 | No named group |
| Applebot | 0 | Allowed by name |
| GoogleOther | 0 | Allowed by name |
How do the search engine crawlers compare?
Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.
Sorted by top sites blocking, high to low.
| Agent | ||||||
|---|---|---|---|---|---|---|
| bingbot | Microsoft | Search indexing | Respected | IP ranges + reverse DNS | Allowed by name | 2.2% |
| Googlebot | Search indexing | Respected | IP ranges + reverse DNS | Allowed by name | 1.9% | |
| Googlebot-Image | Search indexing | Respected | IP ranges + reverse DNS | Follows Googlebot | Not in study | |
| Googlebot-Video | Search indexing | Respected | IP ranges + reverse DNS | Follows Googlebot | Not in study | |
| GoogleOther-Image | Product operations | Respected | IP ranges + reverse DNS | Follows GoogleOther | Not in study | |
| Kagibot | Kagi | Search indexing | Respected | reverse DNS | No named group | Not in study |
| PetalBot | Huawei | Search indexing | Respected | reverse DNS | No named group | Not in study |
| Applebot | Apple | Search indexing | Respected | IP ranges + reverse DNS | Allowed by name | Not in study |
| GoogleOther | Product operations | Respected | IP ranges + reverse DNS | Allowed by name | Not in study |
bingbot
- Operator
- Microsoft
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Allowed by name
- Top sites blocking
- 2.2%
Googlebot
- Operator
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Allowed by name
- Top sites blocking
- 1.9%
Googlebot-Image
- Operator
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Follows Googlebot
- Top sites blocking
- Not in study
Googlebot-Video
- Operator
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Follows Googlebot
- Top sites blocking
- Not in study
GoogleOther-Image
- Operator
- What the fetch is for
- Product operations
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Follows GoogleOther
- Top sites blocking
- Not in study
Kagibot
- Operator
- Kagi
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- reverse DNS
- attensira.com
- No named group
- Top sites blocking
- Not in study
PetalBot
- Operator
- Huawei
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- reverse DNS
- attensira.com
- No named group
- Top sites blocking
- Not in study
Applebot
- Operator
- Apple
- What the fetch is for
- Search indexing
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Allowed by name
- Top sites blocking
- Not in study
GoogleOther
- Operator
- What the fetch is for
- Product operations
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Allowed by name
- Top sites blocking
- Not in study
Each row read from the operator's documentation, 3 Sep 2026 to 4 Sep 2026.
“Not in study” means the token is not one our robots.txt study counts yet, not that no site blocks it.
Which crawlers follow my Googlebot rules?
More than Googlebot. Apple documents that without an Applebot group, Applebot follows your Googlebot rules, and Kagi documents the same fallback for Kagibot. A Disallow written for Google can quietly govern Siri and Spotlight search and Kagi too. Name each crawler you mean.
GoogleOther is not Googlebot. Google describes it as generic product-team crawling and states that preferences addressed to it do not affect any specific product. Googlebot-Image and Googlebot-Video scope Google's media features without touching page indexing.
- Reverse DNS ends in
- *.googlebot.com*.geo.googlebot.com
- IP range file
- common-crawlers.json
Google documents both methods. Verify before you rate-limit: sites let Googlebot through, which makes it a name worth borrowing. A reverse record can be set by whoever controls the address, so follow it with a forward lookup. A spike of "Googlebot" that fails both checks is a scraper, and no robots.txt rule would have reached it. The Googlebot entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Google documentation · Do search engine crawlers matter for AI answers?
Some indexes feed assistants directly. Apple says Applebot powers Siri, Spotlight and Safari search and its data may also help train Apple foundation models. Huawei says Petal Search, which PetalBot indexes, also powers content recommendations in Huawei Assistant and AI Search. Being out of the index is being out of those answers.
Verification differs by operator. Google publishes common-crawlers.json and googlebot.com hostnames, Apple publishes both a range file and applebot.apple.com, Microsoft publishes bingbot.json and documents reverse DNS to search.msn.com, and Kagi lists four addresses on a web page.
- Reverse DNS ends in
- *.aspiegel.com*.petalsearch.com
Huawei publishes no IP range file for PetalBot; verify by the reverse-DNS lookup.
Huawei's written instruction says to verify that the domain name is in aspiegel.com, while the worked example on the same page resolves an address to a petalbot-...petalsearch.com hostname. Both are recorded here because the operator publishes both; the forward lookup back to the original address is the step that actually settles it. The PetalBot entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Huawei documentation · How often do the most visited sites block search engine crawlers?
| Name | Disallowed at the root | Count |
|---|---|---|
| bingbot | 2.2% | 179 |
| Googlebot | 1.9% | 156 |
Web engineer · Writes robots rules like theseRobots.txt access study, our own fetch · · n=8,036 robots.txt filesbingbot is the most blocked here: 179 of 8,036 sites disallow it at the root. The least blocked, Googlebot, is disallowed by 156.
A site that never names an agent is counted as unspecified, not as allowing it.
What else do people ask?
What is a search engine crawler?
Does blocking Googlebot keep my content out of Gemini training?
Does Applebot follow my Googlebot rules?
Does Bing publish an IP list for bingbot?
Where do these facts come from?
11 claims on this page, each with the page it was read from; last checked .
Google documents that only one robots.txt group is valid for a particular crawler. Its crawlers follow the group with the most specific user agent that matches and ignore the others, and user-agent-specific groups are not combined with the * group.
developers.google.com · retrieved
“Only one group is valid for a particular crawler.”
Google documents crawling preferences addressed to the Googlebot-Image user agent as affecting Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented.
developers.google.com · retrieved
“Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented.”
Google documents verifying its crawlers by reverse DNS lookup on the accessing address, checking the domain name is googlebot.com, google.com or googleusercontent.com, then a forward lookup back to the same address; the common crawlers' reverse DNS mask matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com and their ranges are published in common-crawlers.json.
developers.google.com · retrieved
“Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”
Kagi documents Kagibot as the web crawler for the Kagi search engine.
kagi.com · retrieved
“Kagibot is the web crawler for the Kagi search engine.”
Huawei documents PetalBot as an automatic program of the Petal search engine whose function is to access both PC and mobile websites and establish an index database, enabling users to search site content in Petal search engine and presenting content recommendations in Huawei Assistant and AI Search services, both powered by Petal Search.
aspiegel.com · retrieved
“The function of PetalBot is to access both PC and mobile websites and establish an index database which enables users to search the content of your site in Petal search engine”
Apple states that the data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri and Safari, and that it may also be used to help train Apple foundation models powering generative AI features across Apple products.
support.apple.com · retrieved
“The data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari.”
Microsoft documents bingbot as the Bing crawler, announcing its user-agent strings and describing its web page rendering engine as evergreen, kept up to date with the most recent stable version of Microsoft Edge.
blogs.bing.com · retrieved
“regularly updating our web page rendering engine to the most recent stable version of Microsoft Edge”
Microsoft documents verifying bingbot by performing a reverse DNS lookup on the source address to confirm it resolves to a name ending in search.msn.com, then performing a forward DNS lookup on that name to confirm it resolves back to the same address.
blogs.bing.com · retrieved
“Perform a reverse DNS lookup using the IP address from the logs to verify that it resolves to a name that end with search.msn.com”
Microsoft documents that bingbot fetches and analyses the robots.txt file at the root of each host, honours a group addressed to bingbot or msnbot when one exists and the wildcard group otherwise, and keeps the directives in memory for a few hours before fetching the file again.
blogs.bing.com · retrieved
“it keeps your directives in memory for a few hours”
Bing publishes the addresses bingbot crawls from as a JSON range file at www.bing.com/toolbox/bingbot.json. Fetched on 2026-10-05, it carried a creationTime of 2024-01-03T10:00:00 and 28 IPv4 prefixes. This supersedes Microsoft's 2012 statement that Bing did not publish such a list.
bing.com · retrieved
“"creationTime": "2024-01-03T10:00:00.121331"”
Google states that its common crawlers always respect robots.txt rules for automatic crawls.
developers.google.com · retrieved
“always respect robots.txt rules for automatic crawls”
Want whatever it takes to win the answer?
7 days free · card required