Bots
What is Google-Agent, and should you allow it?
Google-Agent is the agent Google documents for software hosted on its infrastructure that navigates the web and performs actions when a user asks. Allow it unless you are ready to turn away people using an assistant. Google says its user-triggered fetchers generally ignore robots.txt, so a Disallow records a preference; stopping it takes a network-level block.
7 days free · card required
0requests filed as Google-Agent · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| Google-Agent (this page) | 0 | Wildcard |
| meta-externalfetcher | 0 | Wildcard |
None of the 2 had a request filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Google-Agent in robots.txt?
Paste this into the robots.txt at the root of your domain. The operator says the agent may not read robots.txt, so the group records your preference; enforce anything sensitive on the server. Beside it, the group attensira.com serves Google-Agent today.
# Google documents user-triggered fetchers as generally# ignoring robots.txt, so read this as a statement of# intent, not an enforcement mechanism. If the fetches# must actually stop, the control is at the network layer.User-agent: Google-AgentDisallow: /# No group names Google-Agent, so it reads the wildcard# group.User-agent: *Allow: /Disallow: /api/We do not name Google-Agent. It falls through to the wildcard group, which allows everything except /api/. Its operator states robots.txt may not apply to it, so this group is a preference at most.
Read the whole file, or write yours with the robots.txt generator.
Who runs Google-Agent, and what is it for?
Google runs it, and we file it under user-triggered fetch. Browses and acts on a site for one person, at that person's request. It is not a crawl and not a training fetch.
From Google’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Why should most sites let Google-Agent in?
Because each visit traces back to a person. Google documents Google-Agent as used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request1. Two words do the work. Navigate: it moves through a site the way a visitor does, not the way an indexer does. Upon user request: somebody asked an agent to do something that involves your page.
That makes it a different decision from every crawler in this directory. Blocking a training crawler refuses a corpus. Blocking Google-Agent refuses a person who chose to use an assistant to get something done on your site. For a business that sells online, that is demand, not extraction.
Google's documentation does not say these fetches produce a citation or a link, so this page does not claim one. The honest reading is engagement: someone is doing something that involves your page right now.
Why doesn't a robots.txt rule stop Google-Agent?
Google states that because the fetch was requested by a user, its user-triggered fetchers generally ignore robots.txt rules3. That is the same reasoning that has always exempted a browser from robots.txt: a person visiting your page is not a crawler, and Google's position is that a person's agent is closer to the browser.
The consequences are practical:
- A Disallow records a preference. Write it if it reflects your intent, and do not rely on it. The block above says so in a comment, so whoever edits the file next is not misled.
- Stopping the fetches is a network or application decision. That means denying a class of visitor, which deserves a deliberate choice rather than a line someone assumes works.
- Do not judge the rule by whether traffic stopped. It was documented not to stop it.
OpenAI says something similar about ChatGPT-User, while Anthropic says Claude-User honours robots.txt. There is no single rule for "AI agents", which is why ChatGPT-User and Claude-User have their own pages.
How should Google-Agent appear in your reports?
In its own row. In the four-way job taxonomy of the AI crawler guide, Google-Agent is user-triggered, and that row behaves unlike the others. A training fetch takes your text and returns nothing measurable. A user-triggered fetch means somebody is currently trying to do something with your page.
Folding Google-Agent into a "bot traffic" total buries the one bucket that is closer to a visit than to extraction. A rise then looks like a cost when it may be demand. If someone proposes classifying it as scraping in a quarterly report, the documented purpose is the answer.
Why does verification matter more for this token?
Because an impersonator inherits the excuse. A scraper calling itself Google-Agent can point to Google's own statement that the agent generally ignores robots.txt. So check the address before you treat any of this traffic as Google's.
Google publishes the addresses its user-triggered agents fetch from at user-triggered-agents.json and documents the reverse-DNS form google-proxy-*.google.com for user-triggered fetchers4. Prefer the range file. Where you use reverse DNS, follow it with a forward lookup back to the same address, since a reverse record can be set by whoever controls the address. If a spike fails both checks, it is not Google, and a network block is both justified and the only thing that will work.
- Reverse DNS ends in
- *.google.com
- IP range file
- user-
triggered- agents.json
ResearcherTranscribed from Google documentation · How should you match Google-Agent in logs?
Google's published strings carry Chrome/W.X.Y.Z as a literal placeholder for the Chrome version, and the identifying fragment is the bare Google-Agent2. Match that token case-insensitively, as a whole word. For a strict match, the Google documentation URL inside the header is a second anchor.
How is Google-Agent different from Google's other fetchers?
Google-Agent
- Who asked for the fetch
- A user, through an agent that navigates and acts
- Robots.txt, per Google
- Generally ignored (user-triggered)
- Who asked for the fetch
- A user who added your URL as a notebook source
- Robots.txt, per Google
- Generally ignored (user-triggered)
Googlebot
- Who asked for the fetch
- Nobody in particular; it builds Search's index
- Robots.txt, per Google
- Respected for automatic crawls
What can Attensira show you about Google-Agent?
AI traffic, on Growth and Business, counts which pages AI agents fetch, grouped by operator and declared purpose, measured at the edge because agents and crawlers do not run your JavaScript. Agent-driven fetches therefore sit apart from crawls. It reports user agents as sent; checking the range file stays your step. How Google's AI answers present your brand is read separately and daily. See crawler logs and the robots.txt generator.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https:/Agents filed beside Google-Agent
Should I allow Google-Agent in my situation?
Concrete cases, and what each one implies for this agent.
You added a Google-Agent Disallow and the fetches continued.
Expect that. If the traffic genuinely has to stop, the control is at the network or application layer, not in robots.txt.
Google documents its user-triggered fetchers as generally ignoring robots.txt rules because the fetch was requested by a user. A robots.txt rule here records a preference rather than enforcing one, and a team that believes otherwise will spend weeks debugging a file that is working exactly as documented.
Google-Agent traffic is rising and someone wants to classify it as scraping in the quarterly report.
Count it as user-initiated engagement, in its own bucket, separate from bulk crawling.
Google documents Google-Agent as used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. Each hit corresponds to a person's request rather than to an indexing sweep, so averaging it in with bulk crawlers hides both signals — and it is the one bucket that behaves more like traffic than like extraction.
A spike of requests claims to be Google-Agent and you suspect a scraper borrowing the name.
Match the source addresses against Google's published user-triggered-agents.json before drawing any conclusion.
Google publishes the addresses its user-triggered agents fetch from as a machine-readable file, which makes this decidable rather than a matter of judgement. It matters more here than for a bulk crawler, because the token is documented as generally ignoring robots.txt and so an impersonator wearing the name inherits the same excuse.
What else do people ask?
What is Google-Agent?
Does Google-Agent obey robots.txt?
Is Google-Agent a crawler or a training bot?
Where do these facts come from?
4 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Google documents Google-Agent as used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.
developers.google.com · retrieved
“Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”
Google documents Google-Agent as the user agent token, with a desktop header string built on a Linux Chrome preamble whose compatible clause names Google-Agent and links to Google's Google-Agent documentation page, with Chrome/W.X.Y.Z as the version placeholder, and a corresponding mobile form.
developers.google.com · retrieved · changes often, check the source
Google states that because the fetch was requested by a user, its user-triggered fetchers generally ignore robots.txt rules.
developers.google.com · retrieved
“Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”
Google documents the reverse-DNS form google-proxy-***-***-***-***.google.com for user-triggered fetchers and publishes the addresses used by its user-triggered agents at developers.google.com/static/crawling/ipranges/user-triggered-agents.json.
developers.google.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required