Bots
Rogerbot: Moz's site audit crawler
Rogerbot is Moz's site audit crawler. It crawls a site because a Moz Pro customer pointed a campaign or on-demand crawl at it. It obeys robots.txt and crawl-delay and feeds no AI answer engine. Moz publishes no IP range, so a visit cannot be verified. Block it only if nobody you work with audits the site in Moz Pro.
7 days free · card required
0requests filed as rogerbot · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| rogerbot (this page) | 0 | Wildcard |
| AhrefsBot | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| DataForSeoBot | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block rogerbot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves rogerbot today.
# Moz advises placing this group above your wildcard (*)# group.User-agent: rogerbotDisallow: / # Or keep the audit crawler and slow it down.# Moz advises against a delay above 30 seconds.# User-agent: rogerbot# Crawl-delay: 10# No group names rogerbot, so it reads the wildcard group.User-agent: *Allow: /Disallow: /api/We do not name rogerbot. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs rogerbot, and what is it for?
Moz runs it, and we file it under product operations. Crawls a site because a Moz Pro customer pointed a campaign or an on-demand audit at it. The results go to that customer, not to any search or AI engine.
From Moz’s documentation, read .
How is an audit crawler different from a link-index crawler?
Moz runs two crawlers with two tokens. Rogerbot audits; Dotbot powers the Moz Link Index behind Link Explorer and the Moz Links API.6 A rule for one does nothing to the other.
The split matters beyond Moz. A link-index crawler fetches your site on the vendor's initiative so any of its customers can look you up. An audit crawler fetches because one customer asked, and only that customer sees the result. Most sites write one rule per vendor name, so a rule meant to keep a database out also stops their own agency's audit.
What crawl-delay should Rogerbot get?
Moz does the arithmetic for you: a 10-second delay allows roughly 8,640 pages a day, and above 30 seconds the crawl might never finish.5 That turns a politeness setting into a capacity calculation. Divide your URL count by the pages a day your delay allows before you set it. A large site under a long delay does not error; its audit keeps arriving incomplete, which reads like a tool problem.
How do I verify that a visit really came from Rogerbot?
You cannot. Moz says it cannot provide an IP address or range because Rogerbot uses thousands of dynamic addresses that change with each crawl, and that the best option available is the rogerbot user agent.4 There is no range file and no reverse DNS suffix, and a header anyone can send identifies nothing. So never grant an unconditional CDN bypass on this token. If your own audit needs an allowlist entry, pair it with a rate limit, so a forged header gets a slow crawl rather than an open door. The methods that do work elsewhere are in verifying a crawler.
Moz publishes no reverse-DNS suffix and no IP range file for rogerbot, so a visit can be matched only by its user agent.
ResearcherTranscribed from Moz documentation · What user-agent string does rogerbot send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
The version in this string changes; check Moz’s page before relying on the exact text.
What does allowing rogerbot cost?
Allowing it costs the load of an audit crawl, which you can slow with crawl-delay. Blocking it breaks Moz Pro audits of your site for anyone using them, including your own team or agency. It costs no search or AI visibility.
rogerbot (robots.txt product token; Moz publishes the partial agent string rogerbot/1.2 rather than a full User-Agent header)Should I allow rogerbot in my situation?
Concrete cases, and what each one implies for this agent.
Your Moz Pro site crawl reports far fewer pages than the site has.
Check for a rogerbot Disallow, and check where your named group sits relative to the wildcard group.
Moz states Rogerbot obeys robots.txt, and advises placing a user-agent record above the wildcard record so bots find it. A crawl that quietly stops short usually reflects a rule somebody wrote to keep audit tools out, not a fault in the tool.
Security asked you to allowlist your SEO tools at the CDN by IP.
Tell them it is not possible for this one, and allowlist on the token with rate limits instead.
Moz states it cannot provide an IP range because Rogerbot uses thousands of dynamic addresses that change each crawl, and that identifying it by user-agent is the best option available. A token-based allowlist is by definition forgeable, so pair it with a rate limit rather than an unconditional bypass.
Rogerbot is crawling too heavily during business hours.
Add a Crawl-delay of 10 for rogerbot; do not set it to 60.
Moz notes that a 10-second delay permits roughly 8,640 pages per day and advises against a delay larger than 30 seconds because the crawl may then never finish. An over-large delay produces a permanently incomplete audit rather than a polite one.
What else do people ask?
Where does Rogerbot send what it crawls?
How is Rogerbot different from Dotbot?
Does Rogerbot obey robots.txt?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Moz documents Rogerbot as the Moz crawler for Moz Pro Campaign site audits, which accesses the code of a site to deliver reports back to the Moz Pro Campaign and serves the Site Crawl report, On-Demand Crawl, Page Optimisation report and On-Page Grader.
moz.com · retrieved
“Rogerbot is the Moz crawler for Moz Pro Campaign site audits.”
Moz identifies the crawler by the robots.txt user-agent token rogerbot and, asked for the full agent string, says to look out for rogerbot/1.2 in logs.
moz.com · retrieved · changes often, check the source
“Keep an eye out for rogerbot/1.2 when digging through your logs.”
Moz states that Rogerbot is built to obey robots.txt files, and advises placing a bot's own user-agent record above the wildcard record so the bot finds it.
moz.com · retrieved
“Rogerbot is built to obey robots.txt files”
Moz states that it cannot provide an IP address or range of IP addresses for Rogerbot because the crawler is designed with a dynamic approach using thousands of dynamic IP addresses that change each crawl, and that the best option available is to identify the crawler by the rogerbot user-agent.
moz.com · retrieved
“The best option available is to identify our crawler by User-agent: rogerbot.”
Moz notes that a Crawl-delay of 10 seconds allows Rogerbot to access about 8,640 pages in a single day, and does not recommend a crawl delay larger than 30 seconds because Rogerbot might not be able to finish the crawl.
moz.com · retrieved
“We don't recommend adding a crawl delay larger than 30 seconds, or rogerbot might not be able to finish the crawl of your site.”
Moz states that Rogerbot is its site audit crawler for Moz Pro campaigns and is different from Dotbot, the web crawler that powers the Moz Link Index.
moz.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required