Bots
What is Googlebot, and should you allow it?
Googlebot is Google's search crawler: it finds and fetches pages to build the index behind Google Search, Discover, Images, Video and News. Allow it on any page you want found. Blocking it removes you from Search and does not opt you out of Gemini training; that is a separate token, Google-Extended, which never fetches anything.
7 days free · card required
189Googlebot requests to attensira.com
to 80 pages · Google Search #6 of 7 operators
| Operator | Share of requests | Requests |
|---|---|---|
| Meta | 131,105 | |
| OpenAI | 7,536 | |
| ByteDance | 3,505 | |
| Perplexity | 621 | |
| Microsoft | 541 | |
| Google Search (this page) | 189 | |
| Anthropic | 104 |
User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP · · n=143,601 requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Googlebot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Googlebot today.
# Googlebot is how you appear in Google Search.# Disallowing it removes you from the index; it is not a# training opt-out.User-agent: GooglebotAllow: / # The training control is a separate token, and it issues# no requests.User-agent: Google-ExtendedDisallow: /User-agent: GooglebotAllow: /Disallow: /api/We name Googlebot and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs Googlebot, and what is it for?
Google runs it, and we file it under search indexing. Fills the index behind Google Search, Discover, Images, Video and News, which Google's AI Overviews are built on. Gemini training is a separate control token, Google-Extended.
From Google’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Does blocking Googlebot opt you out of Gemini training?
No, and confusing the two is costly: blocking Googlebot to refuse training takes you out of Search. Googlebot is the crawler Google documents as finding information for building Google's search indexes, and Google names what that crawling feeds: Search including Discover and all Search features, plus Images, Video and News2. Disallow it and you leave all of them.
Training is a different job with a different token. Google publishes Google-Extended as a standalone product token that manages whether content Google crawls may be used for training future generations of Gemini models, and states that it does not affect a site's inclusion in Google Search and is not a ranking signal6. Few operators separate "you may index me" from "you may not train on me" that cleanly. The robots.txt block above uses the separation: Googlebot allowed, Google-Extended disallowed.
The study figures below show most of the web already understands this. Sites block Google-Extended several times as often as they block Googlebot.
Why can't you treat the two Googlebots differently?
Google documents Googlebot as two crawler types, Smartphone and Desktop, each with its own header string, and both answer to the one robots.txt token Googlebot3. No robots.txt rule can address one without the other. If you want to serve the desktop crawler something different from the mobile one, the place to do that is the page, not the file.
Two more structural points. A rule under User-agent: * catches Googlebot only while Googlebot has no group of its own. And once you write a Googlebot group, Google reads that group alone: its crawlers follow the most specific group that matches and do not combine it with the wildcard1. Any rule you still want Googlebot to obey has to be repeated inside its group. Read the resulting file back.
How do I verify that a visit really came from Googlebot?
A user-agent header is free text, so a log line naming Googlebot is a claim until you check it.
Look up the visiting IP, confirm the hostname ends in one of the suffixes in the record, then look that hostname up and confirm it returns the same IP, or match the IP against the published ranges.
Google documents both methods. Verify before you rate-limit: sites let Googlebot through, which makes it a name worth borrowing. A reverse record can be set by whoever controls the address, so follow it with a forward lookup. A spike of "Googlebot" that fails both checks is a scraper, and no robots.txt rule would have reached it.
What does Googlebot look like in a real site's logs?
On attensira.com, Attensira's own AI traffic measurement recorded 189 Googlebot requests across 80 pages in the 30 days to 4 October 2026. In the same window, Bing's crawler made 541 requests and OpenAI's agents 7,536. Nothing arrived under a separate "AI Overviews" agent, which is what Google's documentation predicts: AI Overviews are a Search feature, built on the index Googlebot fills.
That last point is why Googlebot belongs in an AI crawler directory at all. For Google's AI answers, the crawler that matters is the ordinary search crawler. One site and one window, with user agents as sent, but the pattern is the useful part: a low Googlebot count is not a sign that Google ignores you, and a high OpenAI count is not a sign that ChatGPT cites you.
- Reverse DNS ends in
- *.googlebot.com*.geo.googlebot.com
- IP range file
- common-crawlers.json
ResearcherTranscribed from Google documentation · Agents filed beside Googlebot
How often do sites block Googlebot?
Measured by us, not reported by Google: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow Googlebot at the root
- 1.9%
- 156 of 8,036 domains
- allow it by name
- 6.9%
- 554 domains
- never mention it
- 91.2%
- 7,326 domains
Why match the token and not the version?
Google prints Chrome/W.X.Y.Z in its published strings as a literal placeholder for whichever Chrome version the crawler runs, and Googlebot/2.1 is a version too3. Match the bare token Googlebot, case-insensitively, as a whole word. A filter pinned to a version stops matching after a bump and fails silently: the traffic keeps arriving and your report shows less of it.
How does Googlebot compare with the other Google tokens?
Googlebot
- Fetches pages?
- Yes
- What it decides
- Whether you are in Google Search and its AI features
Google-Extended
- Fetches pages?
- No
- What it decides
- Whether crawled content may train future Gemini models
GoogleOther
- Fetches pages?
- Yes
- What it decides
- Google's internal research and one-off crawls
The full pairing is in Google-Extended vs Googlebot. For how Google's AI answers differ from ChatGPT's search, see Google AI Overviews vs ChatGPT search.
What can Attensira show you about Googlebot?
AI traffic, on Growth and Business, counts which pages crawlers fetch, Googlebot included, measured at the edge because no crawler runs JavaScript. It reports user agents as sent; the reverse DNS check stays yours. The other half is the answer itself: Attensira reads Google AI Overviews every day on every plan, alongside ChatGPT, Perplexity and Claude, and shows whether your pages are the ones cited. See crawler logs and the robots.txt generator.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http:/In this entry
Should I allow Googlebot in my situation?
Concrete cases, and what each one implies for this agent.
You want out of Gemini training but you cannot afford to lose Google Search traffic.
Leave Googlebot allowed. Disallow Google-Extended.
Google documents Google-Extended as the standalone product token that manages whether crawled content may be used for training future generations of Gemini models, and states that it does not impact a site's inclusion in Google Search nor act as a ranking signal. Blocking Googlebot instead does the one thing you said you could not afford.
A traffic spike claiming to be Googlebot is saturating your origin and you are about to rate-limit it.
Verify the source addresses first — reverse DNS to googlebot.com with a forward lookup back, or a match against common-crawlers.json.
Sites make exceptions for Googlebot, which makes its name worth copying. If the addresses do not verify, you are looking at a scraper wearing Google's name, and a robots.txt rule would not have slowed it by one request.
Your mobile pages differ from your desktop pages and you want to serve the desktop crawler something else.
You cannot separate them in robots.txt. Fix the content instead.
Google documents Googlebot Smartphone and Googlebot Desktop as two crawler types under the single robots.txt token Googlebot, so no rule can address one without the other. Any difference has to be handled in what you serve, not in what you allow.
What else do people ask?
What is Googlebot?
Does blocking Googlebot stop Google training Gemini on my content?
Can I block Googlebot Smartphone but not Googlebot Desktop?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Google documents that only one robots.txt group is valid for a particular crawler. Its crawlers follow the group with the most specific user agent that matches and ignore the others, and user-agent-specific groups are not combined with the * group.
developers.google.com · retrieved
“Only one group is valid for a particular crawler.”
Google documents Googlebot as finding information for building Google's search indexes, affecting Google Search including Discover and all Google Search features, as well as Google Images, Google Video and Google News.
developers.google.com · retrieved
“Find information for building Google's search indexes”
Google documents Googlebot as two crawler types, Smartphone and Desktop, each with its own full header string carrying "Googlebot/2.1; +http://www.google.com/bot.html" and a Chrome/W.X.Y.Z version placeholder, both addressed by the robots.txt token Googlebot.
developers.google.com · retrieved · changes often, check the source
Google states that its common crawlers always respect robots.txt rules for automatic crawls.
developers.google.com · retrieved
“always respect robots.txt rules for automatic crawls”
Google documents verifying Googlebot by reverse DNS lookup to a hostname ending in googlebot.com or geo.googlebot.com followed by a forward lookup back to the same address, or by matching the address against its published common-crawlers.json range file.
developers.google.com · retrieved
“Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.”
Google documents Google-Extended as a standalone product token publishers use to manage whether content Google crawls may be used for training future generations of Gemini models, and states that it does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
developers.google.com · retrieved
“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search”
Want whatever it takes to win the answer?
7 days free · card required