Bots
Mediapartners-Google: the AdSense crawler
Mediapartners-Google is Google's AdSense crawler. It visits pages on sites that show AdSense ads so Google can serve ads relevant to the content. It has no documented effect on Google Search or AI answers. Google says it ignores the wildcard robots.txt group, so only a group naming Mediapartners-Google reaches it. Block it only if you have no AdSense.
7 days free · card required
0requests filed as Mediapartners-Google · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| Mediapartners-Google (this page) | 0 | Not reached |
| AdsBot-Google | 0 | Not reached |
| AdsBot-Google-Mobile | 0 | Not reached |
| OAI-AdsBot | 0 | Wildcard |
None of the 4 had a request filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Mediapartners-Google in robots.txt?
Paste this into the robots.txt at the root of your domain. Only a group that names the token reaches it; your * group does not. Beside it, the group attensira.com serves Mediapartners-Google today.
# A wildcard group does not reach this agent. Name the# token.User-agent: Mediapartners-GoogleDisallow: /members-only/ # Blocking it entirely is an AdSense decision about your# own inventory.it ignores the User-agent: * group
We do not name Mediapartners-Google, and its operator states it ignores the User-agent: * group, so no rule in our file reaches it. Only a group that names Mediapartners-Google would.
Read the whole file, or write yours with the robots.txt generator.
Who runs Mediapartners-Google, and what is it for?
Google runs it, and we file it under product operations. Visits pages on sites that run AdSense so Google can show ads relevant to them; its rules affect AdSense and Ad Manager. It has no documented effect on Search or AI answers.
From Google’s documentation, read .
Whose pages does Mediapartners-Google read?
Yours, if you run AdSense. Google documents the crawler as visiting participating sites in order to provide them with relevant ads, and says rules addressed to it affect AdSense and Ad Manager.1
That sentence contains the whole decision. This crawl is how a page and an ad slot get matched. Its purpose is not to find you, rank you or learn from you; it reads the page an ad will sit on, on a site that has already chosen to carry ads.
Is Mediapartners-Google the same as AdsBot-Google?
No. Google documents AdsBot-Google separately, as affecting Google Ads' ability to check web page ad quality.4 The two tokens sit three rows apart in the same table and are constantly conflated.
The clean way to hold them apart is to ask whose page is being read. AdsBot reads an advertiser's landing page, to check the quality of a destination someone is paying to send traffic to. Mediapartners-Google reads a publisher's page to work out which ads belong on it. Same company, opposite ends of the transaction, and a robots.txt rule aimed at one has no effect on the other.
How do I verify that a visit really came from Mediapartners-Google?
A user-agent header is free text, so a log line naming Mediapartners-Google is a claim until you check it.
Look up the visiting IP, confirm the hostname ends in one of the suffixes in the record, then look that hostname up and confirm it returns the same IP, or match the IP against the published ranges.
A special-case crawler, so it resolves under the rate-limited-proxy mask on google.com rather than googlebot.com, and appears in special-crawlers.json rather than common-crawlers.json.
Why does a wildcard robots.txt rule not reach it?
Google states that the global user agent (*) is ignored for this crawler.3 The reasoning is in Google's framing of the whole special-case category: these are agents operating where there is an agreement between the crawled site and the product. Running AdSense is that agreement.
So the usual sequence (write User-agent: * with Disallow: /, watch this agent keep arriving, conclude something is impersonating Google) describes documented behaviour. It also means attensira.com's own Disallow: /api/, which sits in our * group, does not apply to it. If you want it gone from a path, name the token.
Why do AdSense ads stop fitting the page after a robots.txt cleanup?
Because pages this crawler cannot read are pages AdSense cannot match to an ad, and the failure is silent: no error, no warning, just ads that stop fitting the page they are on. If contextual relevance degraded after someone tidied robots.txt, look for a group naming this token first.
Blocking a members-only area from it is a reasonable narrower move, and the block above shows that shape. The general rules for per-token groups are in the robots.txt chapter.
- Reverse DNS ends in
- *.google.com
- IP range file
- special-crawlers.json
ResearcherTranscribed from Google documentation · Agents filed beside Mediapartners-Google
Which user-agent headers does it send?
Google publishes two forms: a desktop agent that is the bare string Mediapartners-Google, and a mobile agent built from a device string followed by (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html).2
Because the desktop form has no browser shape at all, request filters that expect one discard it. Because it is twenty characters, anything can send it. Both facts argue for the same thing: match the bare token Mediapartners-Google case-insensitively for logging, and decide access on a verified address.
Why does a Googlebot check reject the real AdSense crawler?
It is a special-case crawler, so the Googlebot verification recipe does not apply. Google documents the reverse DNS mask for this category as rate-limited-proxy-***-***-***-***.google.com and publishes the ranges in special-crawlers.json.5
Reverse-resolve the accessing address, confirm the domain, then forward-resolve the hostname and check it returns the same address. An allowlist keyed to googlebot.com will reject the real AdSense crawler and admit any forgery that copies the header, which is precisely backwards. The full procedure is in verifying a crawler.
What does allowing Mediapartners-Google cost?
If your site shows AdSense ads, allowing it is how Google matches those ads to your pages, so blocking it works against your own inventory. If you run no AdSense, it should rarely visit and blocking it changes nothing. Either way it has no documented effect on Google Search or AI answers.
Mediapartners-GoogleIn this entry
- Whose pages does Mediapartners-Google read?
- Is Mediapartners-Google the same as AdsBot-Google?
- Why does a wildcard robots.txt rule not reach it?
- Why do AdSense ads stop fitting the page after a robots.txt cleanup?
- Which user-agent headers does it send?
- Why does a Googlebot check reject the real AdSense crawler?
Should I allow Mediapartners-Google in my situation?
Concrete cases, and what each one implies for this agent.
Your ads went generic or stopped matching page content after a robots.txt change.
Check whether a rule now names Mediapartners-Google, and restore access to the pages carrying ads.
Google describes the AdSense crawler as visiting participating sites in order to provide them with relevant ads. If it cannot read a page, the mechanism that makes an ad relevant to that page has nothing to work from — and the symptom shows up as revenue rather than as an error.
You wrote a wildcard disallow to keep every bot out of a members-only area and Mediapartners-Google kept reading it.
Write a group addressed to Mediapartners-Google by name for that path.
Google states that the global user agent is ignored for this crawler, so a wildcard group never applied to it. The behaviour you are seeing is documented, and the fix is a named group rather than a stronger wildcard.
Your log parser lists a client whose entire user agent is the single word Mediapartners-Google and flags it as malformed.
Accept it, and verify by address rather than by header shape.
Google publishes the desktop agent as exactly that bare token, with no Mozilla preamble and no version. Filters that require a browser-shaped header treat it as junk, while a scraper can reproduce those twenty characters trivially — so the header tells you almost nothing in either direction and the reverse-DNS round trip tells you everything.
What else do people ask?
What is Mediapartners-Google?
Does Mediapartners-Google follow robots.txt?
What happens if I block Mediapartners-Google?
Where do these facts come from?
5 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Google documents that the Mediapartners-Google crawler visits participating sites in order to provide them with relevant ads, and that crawling preferences addressed to it affect ad-related Google products such as Google AdSense and Google Ad Manager.
developers.google.com · retrieved
“The Mediapartners-Google crawler visits participating sites in order to provide them with relevant ads.”
Google publishes Mediapartners-Google's desktop agent as "Mediapartners-Google" and its mobile agent as a device string followed by "(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)".
developers.google.com · retrieved
“(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)”
Google documents Mediapartners-Google as the robots.txt user agent token for the AdSense crawler and states that the global user agent (*) is ignored.
developers.google.com · retrieved
“The global user agent (*) is ignored.”
Google documents AdsBot-Google as a separate token whose crawling preferences affect Google Ads' ability to check web page ad quality.
developers.google.com · retrieved
“affect Google Ads' ability to check”
Google documents the special-case crawlers' reverse DNS mask as rate-limited-proxy-***-***-***-***.google.com and publishes their IP ranges in the special-crawlers.json object.
developers.google.com · retrieved
“The special-case crawlers' reverse DNS mask matches rate-limited-proxy-***-***-***-***.google.com”
Want whatever it takes to win the answer?
7 days free · card required