Bots
What is dotbot, and should you allow it?
dotbot is Moz's web crawler for the Moz Link Index, which Link Explorer and Moz Pro draw on. It is not an AI crawler, so the choice is commercial. Allow it, scope it to cheap paths, or block it. Moz reads robots.txt only when it first meets your site in a new index crawl, so changes apply late.
7 days free · card required
0AI-view requests from dotbot · 30 days
filed as conventional traffic (SEO tools, uptime checks, link unfurlers)
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| dotbot (this page) | not counted | Wildcard |
| AhrefsBot | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| DataForSeoBot | 0 | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| rogerbot | 0 | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP; it files dotbot as conventional traffic, off its AI view · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block dotbot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves dotbot today.
# Keep the Moz link index out of the whole site.User-agent: dotbotDisallow: / # Or out of specific areas only.# User-agent: dotbot# Disallow: /admin/# Disallow: /scripts/ # Or keep it and slow it down.# User-agent: dotbot# Crawl-delay: 10# No group names dotbot, so it reads the wildcard group.User-agent: *Allow: /Disallow: /api/We do not name dotbot. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs dotbot, and what is it for?
Moz runs it, and we file it under product operations. Crawls for an SEO tool: a link index, a site audit or a test fetch. It feeds no AI answer and, apart from AhrefsBot's work for Yep, no search engine's index.
From Moz’s documentation, read .
Why is dotbot still crawling after you blocked it?
Because Moz reads your robots.txt rarely, by design. Moz states that dotbot only looks at a site's robots.txt the first time it encounters that site during a new index crawl, so a change in permission is not picked up until the next time Moz locates links to the site4.
Moz's robots.txt window is not measured in time at all; it is measured in index cycles, and its length depends on when Moz next finds a link to you. That produces a predictable mistake. Someone adds a dotbot Disallow, sees dotbot in the logs a week later, concludes it ignores robots.txt and escalates to a firewall rule, which also blocks the robots.txt fetch that would eventually have delivered the message.
Write the rule, note the date, and give it an index cycle. Moz states that it always respects the standard Robots Exclusion Protocol for dotbot3; the delay is latency, not defiance.
Should you block dotbot or scope it?
Scope it first. dotbot is Moz's crawler for the Moz Link Index, with the data available in Moz Pro, Link Explorer and the Moz Links API1. It is not an AI crawler: nothing it fetches is described as going into a model or into an assistant's answer. The decision is about who holds data on your backlink profile.
Moz documents three controls, all in the robots.txt block above: a whole-site Disallow, path-level Disallows and Crawl-delay. A site that blocks dotbot outright often objects to the crawl load, not to being in a link index. They objected to a crawler grinding through faceted search or an expensive rendering path at night. Disallowing those paths keeps the link profile intact and removes the cost. Use the whole-site rule only when the objection is to the dataset itself, and in that case decide for the whole category of backlink crawlers, not for Moz alone.
How is dotbot different from Rogerbot?
Moz states dotbot is different from Rogerbot, its site audit crawler for Moz Pro campaigns5.
Job
- dotbot
- Builds the Moz Link Index
- Rogerbot
- Audits a site for a Moz Pro campaign
Who starts it
- dotbot
- Moz, on its own schedule
- Rogerbot
- A Moz customer pointing it at a site
Typical reason to block
- dotbot
- You object to being in the link dataset
- Rogerbot
- Rarely a good idea if your team uses Moz Pro
The same split exists at Ahrefs, between AhrefsBot and its audit crawler. A rule for one never covers the other.
What do you do with a crawler you cannot verify?
Treat its name as a claim. Moz offers the user-agent token as dotbot's only identifier6, so two rules follow. Do not build a CDN allowlist exception around the dotbot token, because anything can send it. And count dotbot hits as claimed rather than confirmed when you report on them. Among backlink crawlers, DataForSeoBot and AhrefsBot publish verification methods; dotbot and SemrushBot do not.
Moz publishes no reverse-DNS suffix and no IP range file for dotbot, so a visit can be matched only by its user agent.
ResearcherTranscribed from Moz documentation · How should you match dotbot in logs?
Moz states the user agent dotbot identifies its crawler and documents the robots.txt token as dotbot2. It publishes no full header string, so there is nothing longer to match. Match the bare token dotbot, case-insensitively, as a whole word. A substring test for "dot" across a bot list collides with things that are not this crawler.
What can Attensira show you about dotbot?
Mainly that it does not belong in your AI numbers. AI traffic, on Growth and Business, groups crawler fetches by operator and declared purpose at the edge, so link-index crawling stays apart from the agents that affect AI answers. It reports user agents as sent, which for dotbot is all anyone can report. See crawler logs and the robots.txt generator.
dotbotAgents filed beside dotbot
Should I allow dotbot in my situation?
Concrete cases, and what each one implies for this agent.
You added a Disallow for dotbot last week and it is still crawling.
Wait for the next index crawl before concluding the rule was ignored.
Moz states that dotbot only looks at robots.txt the first time it encounters a site during a new index crawl, so a change in permission is not seen until Moz next locates links to your site. This is documented behaviour, not a compliance failure, and it is unusually long-lived because it is measured in index cycles, not hours.
You want out of link databases generally, and you have already written rules for AhrefsBot and SemrushBot.
Add dotbot and MJ12bot to the same set — and check whether you meant to include the audit crawlers too.
Each vendor in this cluster runs a separate index crawler under its own token, and none of them is covered by a rule naming another. Moz's index crawler is dotbot; its audit crawler is Rogerbot, and if anyone on your side runs Moz Pro audits you want the first blocked and the second allowed.
dotbot is crawling an area of the site that is expensive to render but worthless as a link target.
Disallow the paths rather than the bot.
Moz documents path-level Disallow for dotbot as a first-class option, with admin, scripts and image directories as its own example. Scoping the rule keeps your link profile in the index while removing the cost.
What else do people ask?
I added a Disallow for dotbot and it is still crawling. Is it ignoring robots.txt?
What is the difference between dotbot and Rogerbot?
Does dotbot support Crawl-delay?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Moz documents dotbot as its web crawler that gathers web data for the Moz Link Index, with the collected data available in the Links section of a Moz Pro campaign, in Link Explorer, and through the Moz Links API.
moz.com · retrieved
“Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index.”
Moz states that the user-agent dotbot is used to identify its crawler, and documents the robots.txt token as dotbot.
moz.com · retrieved
“User-agent: dotbot”
Moz states that it always respects the standard Robots Exclusion Protocol for dotbot, and documents both whole-site and path-level Disallow rules as well as Crawl-delay.
moz.com · retrieved
“we always respect the standard Robots Exclusion Protocol (aka robots.txt)”
Moz states that dotbot only looks at a site's robots.txt file the first time it encounters that site during a new index crawl, so a change in permission is not picked up until the next time Moz locates links to the site.
moz.com · retrieved
“Dotbot only looks at your robots.txt file the first time it encounters that site during a new index crawl.”
Moz states that dotbot is different from Rogerbot, which is its site audit crawler for Moz Pro Campaigns.
moz.com · retrieved
Moz's dotbot documentation publishes no IP ranges and no reverse DNS suffix for the crawler, offering the user-agent token as the only identifier.
moz.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required