Bots
SEO crawlers
SEO bots crawl for SEO tools: link indexes such as AhrefsBot, SemrushBot, MJ12bot, Moz's Dotbot and DataForSeoBot, site auditors such as AhrefsSiteAudit and Rogerbot, Google's Search Console tester and the Screaming Frog spider. None feeds Google, Bing or an AI answer, so blocking one costs that tool's view of your site; blocking AhrefsBot also drops you from Yep.
7 days free · card required
- agents documented
- 9
- publish IP ranges or reverse DNS
- 4 of 9
- requests to attensira.com, 30 days
- 0
- of n=143,601 · 5 not counted
One of the category's rules, from the AhrefsSiteAudit entry, per the operator's documentation; the other 8 are on their own entries.
User-agent: AhrefsSiteAuditAllow: /
ResearcherOperator documentation and our AI traffic board (Attensira MCP) · Which SEO crawlers are documented?
Crawls for an SEO tool: a link index, a site audit or a test fetch. It feeds no AI answer and, apart from AhrefsBot's work for Yep, no search engine's index.
- AhrefsSiteAudit AhrefsSiteAudit crawls a site when someone runs an Ahrefs Site Audit on it. Why blocking it only blinds audits, its 30-URL-a-minute cap, and verification. Ahrefs · Product operations
- Google-InspectionTool Google-InspectionTool is the fetcher behind URL Inspection and the Rich Result Test. Why to allow it, and why Search Console says it cannot fetch. Google · Product operations
- MJ12bot MJ12bot maps links for Majestic through a distributed volunteer network. Its robots.txt and crawl-delay rules, and the one way to verify a visit. Majestic · Product operations
- rogerbot Rogerbot runs Moz Pro site audits for whoever pointed it at your site. Its robots.txt and crawl-delay limits, and why Moz says it cannot be verified. Moz · Product operations
- Screaming Frog SEO Spider The Screaming Frog token means a person is auditing your site from their own machine. How to block it, why it cannot be verified, and the 403 fix. Screaming Frog · Product operations
- SemrushBot SemrushBot is a family of ten robots.txt tokens, one per Semrush tool. Why one rule misses nine of them, why it cannot be verified, and why we block it. Semrush · Product operations
- AhrefsBot AhrefsBot builds the Ahrefs link index and the Yep search engine. Not an AI crawler. How it compares with SemrushBot, and why attensira.com blocks both. Ahrefs · Product operations
- DataForSeoBot DataForSeoBot collects backlink data for DataForSEO. Not an AI crawler. Its 5-second default rate, the crawl-delay control, and reverse DNS verification. DataForSEO · Product operations
- dotbot dotbot is Moz's crawler for the Moz Link Index. Why a new Disallow can take an index cycle to apply, how to scope it, and why it cannot be verified. Moz · Product operations
Which SEO crawlers fetched attensira.com in the last 30 days?
The AI traffic board does not measure SEO tools, uptime checks or link unfurlers, so the 5 rows marked "not counted" have no count to show. Of the 4 it does measure, none had requests filed on attensira.com in the window.
| Agent | Requests | Our robots.txt |
|---|---|---|
| AhrefsSiteAudit | 0 | No named group |
| Google-InspectionTool | 0 | Follows Googlebot |
| MJ12bot | not counted | No named group |
| rogerbot | 0 | No named group |
| Screaming Frog SEO Spider | not counted | No named group |
| SemrushBot | not counted | Blocked site-wide |
| AhrefsBot | not counted | Blocked site-wide |
| DataForSeoBot | 0 | No named group |
| dotbot | not counted | No named group |
How do the SEO crawlers compare?
Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.
Sorted by verify by, low to high.
| Agent | ||||
|---|---|---|---|---|
| AhrefsSiteAudit | Ahrefs | Respected, one exception | IP ranges + reverse DNS | No named group |
| Google-InspectionTool | Respected | IP ranges + reverse DNS | Follows Googlebot | |
| AhrefsBot | Ahrefs | Respected | IP ranges + reverse DNS | Blocked site-wide |
| MJ12bot | Majestic | Respected | Nothing published | No named group |
| rogerbot | Moz | Respected | Nothing published | No named group |
| Screaming Frog SEO Spider | Screaming Frog | Respected by default | Nothing published | No named group |
| SemrushBot | Semrush | Respected | Nothing published | Blocked site-wide |
| dotbot | Moz | Respected | Nothing published | No named group |
| DataForSeoBot | DataForSEO | Respected | reverse DNS | No named group |
AhrefsSiteAudit
- Operator
- Ahrefs
- robots.txt
- Respected, one exception
- Verify by
- IP ranges + reverse DNS
- attensira.com
- No named group
Google-InspectionTool
- Operator
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Follows Googlebot
AhrefsBot
- Operator
- Ahrefs
- robots.txt
- Respected
- Verify by
- IP ranges + reverse DNS
- attensira.com
- Blocked site-wide
MJ12bot
- Operator
- Majestic
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- No named group
rogerbot
- Operator
- Moz
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- No named group
Screaming Frog SEO Spider
- Operator
- Screaming Frog
- robots.txt
- Respected by default
- Verify by
- Nothing published
- attensira.com
- No named group
SemrushBot
- Operator
- Semrush
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- Blocked site-wide
dotbot
- Operator
- Moz
- robots.txt
- Respected
- Verify by
- Nothing published
- attensira.com
- No named group
DataForSeoBot
- Operator
- DataForSEO
- robots.txt
- Respected
- Verify by
- reverse DNS
- attensira.com
- No named group
Each row read from the operator's documentation, 3 Sep 2026.
Should I block SEO crawlers?
Split them by who reads the data. Link-index crawlers such as AhrefsBot, SemrushBot, MJ12bot and Dotbot collect your site into databases their customers query, including your competitors. Audit crawlers such as AhrefsSiteAudit, Rogerbot and Screaming Frog are pointed at a site by someone who wants to fix it, usually you or your agency.
attensira.com blocks AhrefsBot and SemrushBot site-wide and leaves the rest alone; the matrix above reads that from our live robots.txt. Google-InspectionTool is the one to keep: it is what fetches your page when you press Test in Search Console, and Google says preferences addressed to it have no effect on Google Search, so disallowing it only blinds your own tooling.
- Reverse DNS ends in
- *.googlebot.com*.geo.googlebot.com
- IP range file
- common-crawlers.json
One of Google's common crawlers, so it verifies the way Googlebot does. Do it deliberately if you run a strict allowlist: the short desktop header is exactly the shape heuristic bot filters reject, and what you admit should be the verified agent, not its header. The Google-InspectionTool entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Google documentation · Why does blocking Semrush take ten groups?
Semrush uses one robots.txt token per tool, from SemrushBot and SiteAuditBot to SplitSignalBot and RyteBot. A group naming SemrushBot reaches the link crawler only.
Most link crawlers can be slowed rather than blocked. Majestic honors a crawl-delay up to 20 seconds, Moz advises against a Rogerbot delay above 30 seconds, and DataForSEO documents a default of one request every 5 seconds.
Few can be verified. Ahrefs publishes IP ranges and reverse DNS, DataForSEO a reverse DNS mask, and Google-InspectionTool verifies like Googlebot. Moz, Semrush and Majestic publish no ranges, and Screaming Frog runs on its user's own machine with a configurable user agent.
Semrush publishes no reverse-DNS suffix and no IP range file for SemrushBot, so a visit can be matched only by its user agent.
The SemrushBot entry, and the method step by step: how to verify a crawler.
ResearcherTranscribed from Semrush documentation · What else do people ask?
What are SEO bots?
Does blocking AhrefsBot or SemrushBot hurt my rankings?
Why does attensira.com block AhrefsBot and SemrushBot?
Should I block Google-InspectionTool?
Where do these facts come from?
12 claims on this page, each with the page it was read from; last checked .
Ahrefs documents AhrefsSiteAudit as the crawler that powers its Site Audit tool, which checks websites for technical and on-page issues such as broken links and slow pages.
ahrefs.com · retrieved
“AhrefsSiteAudit powers our Site Audit tool.”
Google documents that only one robots.txt group is valid for a particular crawler. Its crawlers follow the group with the most specific user agent that matches and ignore the others, and user-agent-specific groups are not combined with the * group.
developers.google.com · retrieved
“Only one group is valid for a particular crawler.”
Google documents that crawling preferences addressed to the Google-InspectionTool user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console, and that it has no effect on Google Search or other products.
developers.google.com · retrieved
“Crawling preferences addressed to the Google-InspectionTool user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console.”
Google states that its common crawlers always respect robots.txt rules for automatic crawls.
developers.google.com · retrieved
“always respect robots.txt rules for automatic crawls”
Google documents verifying its common crawlers by reverse DNS lookup to a hostname ending in googlebot.com or geo.googlebot.com followed by a forward lookup back to the same address, or by matching the address against its published common-crawlers.json range file.
developers.google.com · retrieved
Majestic describes itself as a UK-based specialist search engine used to paint a map of the internet independent of the consumer-based search engines, and states that MJ12bot maps the link relationships between websites rather than caching web content or personal data.
mj12bot.com · retrieved
“MJ12Bot does not currently cache web content or personal data. Instead it maps the link relationships between websites to build a search engine.”
Moz documents Rogerbot as the Moz crawler for Moz Pro Campaign site audits, which accesses the code of a site to deliver reports back to the Moz Pro Campaign and serves the Site Crawl report, On-Demand Crawl, Page Optimisation report and On-Page Grader.
moz.com · retrieved
“Rogerbot is the Moz crawler for Moz Pro Campaign site audits.”
Screaming Frog documents the SEO Spider as a crawler whose requests are made by the person running the application, with configurable user agent and robots.txt behaviour, used to crawl a site for SEO analysis.
screamingfrog.co.uk · retrieved
“The user-agent configuration allows you to switch the user-agent of the HTTP requests made by the SEO Spider and which robots.txt directives are followed.”
Screaming Frog states that by default the SEO Spider makes requests using its own Screaming Frog SEO Spider user-agent string, and that its user agent is "Screaming Frog SEO Spider" for the purposes of a robots.txt rule.
screamingfrog.co.uk · retrieved
“Its user agent is 'Screaming Frog SEO Spider' so you can include the following in your robots.txt if you wish the Spider not to crawl your site”
Semrush describes SemrushBot as the search bot software that Semrush sends out to discover and collect new and updated web data, feeding tools including Backlink Analytics, Site Audit, Backlink Audit and Link Building.
semrush.com · retrieved
“SemrushBot is the search bot software that Semrush sends out to discover and collect new and updated web data.”
DataForSEO documents DataForSeoBot as constantly crawling the web to collect backlink data for its products, and describes it as belonging to the category of good bots that support robots.txt settings, explicitly identify themselves, and have an official moniker.
dataforseo.com · retrieved
“DataForSEO Bot belongs to the category of Good Bots, meaning it won’t harm your website in any way. Our bot supports your robots.txt settings, explicitely identifies itself and has an official moniker.”
Moz documents dotbot as its web crawler that gathers web data for the Moz Link Index, with the collected data available in the Links section of a Moz Pro campaign, in Link Explorer, and through the Moz Links API.
moz.com · retrieved
“Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index.”
Want whatever it takes to win the answer?
7 days free · card required