Bots
What is DataForSeoBot, and should you allow it?
DataForSeoBot is DataForSEO's crawler: it constantly crawls the web to collect backlink data for its products. It is not an AI crawler, so the decision is commercial. Allow it if you do not mind being in another link dataset; block it, or slow it with Crawl-delay, if you do. Its published reverse DNS mask lets you verify visits.
7 days free · card required
0requests filed as DataForSeoBot · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| DataForSeoBot (this page) | 0 | Wildcard |
| AhrefsBot | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| rogerbot | 0 | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block DataForSeoBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves DataForSeoBot today.
# Block the whole site.User-agent: DataForSeoBotDisallow: / # Or keep it and slow it down. The documented default is# one request every 5 seconds.# User-agent: DataForSeoBot# Crawl-delay: 10# No group names DataForSeoBot, so it reads the wildcard# group.User-agent: *Allow: /Disallow: /api/We do not name DataForSeoBot. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs DataForSeoBot, and what is it for?
DataForSEO runs it, and we file it under product operations. Crawls for an SEO tool: a link index, a site audit or a test fetch. It feeds no AI answer and, apart from AhrefsBot's work for Yep, no search engine's index.
From DataForSEO’s documentation, read .
Is DataForSeoBot something you need to decide about?
Only as part of a decision about backlink crawlers in general. DataForSEO documents DataForSeoBot as constantly crawling the web to collect backlink data for its products, and describes it as belonging to the category of good bots: ones that support robots.txt, identify themselves explicitly and have an official name1.
Functionally it is the same kind of crawler as AhrefsBot, SemrushBot, MJ12bot and Dotbot: it builds a link index other people query about your site.
It is not an AI crawler. Nothing in DataForSEO's description says it feeds a model or an assistant's answer. So the question is commercial. Blocking DataForSeoBot while allowing the other four does not keep your link profile out of anyone's product; it removes you from one dataset. Either the category bothers you, and you write all five groups, or it does not, and this is a well-behaved visitor that costs you little. attensira.com, for what it is worth, blocks AhrefsBot and SemrushBot and leaves the rest.
How do you slow DataForSeoBot down instead of blocking it?
With Crawl-delay, which it honours. DataForSEO states the bot obeys robots.txt and obeys crawl delay, and documents a default visit frequency of one request every 5 seconds that a Crawl-delay directive changes3.
The published default is more useful than it looks. It gives you a baseline, so a burst well above one request every 5 seconds is a signal rather than a vague suspicion. In numbers: one request every 5 seconds is at most 12 a minute, 720 an hour and 17,280 a day for a site it is crawling continuously. Crawl-delay: 10 caps that at 8,640 a day. If your logs show DataForSeoBot well above 12 requests a minute, either the default changed or the traffic is not DataForSEO, and the reverse lookup settles which.
The robots.txt block above covers both decisions: the whole-site block, and a commented Crawl-delay: 10 variant that halves the default rate. DataForSEO also documents a path-level form, which is the better answer when your objection is one expensive section rather than the crawler.
Which backlink crawlers can you actually verify?
Two of the four in this cluster. DataForSEO publishes, on one page, the full user-agent string, the robots.txt token, a yes on robots.txt and crawl-delay, the default crawl frequency, a reverse DNS mask with a worked example, and the crawler's IPv4 and IPv6 subnets4.
DataForSeoBot
- IP list published
- Yes, IPv4 and IPv6 subnets
- Reverse DNS published
- Yes,
crawling-gateway- *. dataforseo. com
- IP list published
- Yes
- Reverse DNS published
- Yes,
ahrefs.comorahrefs.net
- IP list published
- No
- Reverse DNS published
- No
Dotbot (Moz)
- IP list published
- No
- Reverse DNS published
- No
Size does not predict documentation quality here. For the two that publish nothing, a header is a claim you cannot check, so they never belong in a CDN allowlist exception.
- Reverse DNS ends in
- *.dataforseo.com
DataForSEO publishes no IP range file for DataForSeoBot; verify by the reverse-DNS lookup.
ResearcherTranscribed from DataForSEO documentation · Agents filed beside DataForSeoBot
What is easy to get wrong in the user-agent string?
DataForSEO publishes it as Mozilla/5.0 (compatible; DataForSeoBot; +https://dataforseo.com/…), ending in the URL of its bot page, shown in full in the user-agent section above2. Two details trip up filters. There is a space between DataForSeoBot; and the + URL, and a filter written without it will not match. And the casing is unusual: capital D, F and S with a lower-case eo.
There is no version inside the string, which is rare. That is not a promise of stability, so still match the bare token DataForSeoBot, case-insensitively, as a whole word.
What can Attensira show you about DataForSeoBot?
That it is outside your AI picture. AI traffic, on Growth and Business, groups crawler fetches by operator and declared purpose at the edge, so link-database crawlers do not distort the numbers you read for AI agents. It reports user agents as sent; the reverse DNS check stays yours. See crawler logs, and write the backlink-crawler groups with the robots.txt generator.
Mozilla/5.0 (compatible; DataForSeoBot; +https:/Should I allow DataForSeoBot in my situation?
Concrete cases, and what each one implies for this agent.
An unfamiliar token is in your logs and you want to know in one step whether it is genuine.
Run a reverse DNS lookup on the source address and check for the crawling-gateway pattern under dataforseo.com.
DataForSEO publishes the mask crawling-gateway-*.dataforseo.com. A reverse lookup that resolves into that pattern, forward-confirmed, settles the question without downloading a range file — and a lookup that does not is a scraper borrowing the name.
The crawl is welcome but arrives faster than your origin likes.
Set a Crawl-delay for DataForSeoBot instead of blocking it.
DataForSEO documents a default frequency of one visit every 5 seconds and states that the bot obeys crawl delay, so the rate is a parameter you can set rather than a fixed cost you have to accept or refuse.
You have already disallowed AhrefsBot and SemrushBot and want a consistent position on backlink crawlers.
Extend the same rule to DataForSeoBot, MJ12bot and dotbot.
All five collect link data for commercial products, and each is addressed by its own token, so a rule for one reaches none of the others. Deciding the category once and writing five groups is more coherent than blocking whichever name you happened to recognise in the logs.
What else do people ask?
What is DataForSeoBot?
How fast does DataForSeoBot crawl?
Does DataForSeoBot respect robots.txt?
Where do these facts come from?
4 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
DataForSEO documents DataForSeoBot as constantly crawling the web to collect backlink data for its products, and describes it as belonging to the category of good bots that support robots.txt settings, explicitly identify themselves, and have an official moniker.
dataforseo.com · retrieved
“DataForSEO Bot belongs to the category of Good Bots, meaning it won’t harm your website in any way. Our bot supports your robots.txt settings, explicitely identifies itself and has an official moniker.”
DataForSEO publishes DataForSeoBot's user-agent string as "Mozilla/5.0 (compatible; DataForSeoBot; +https://dataforseo.com/dataforseo-bot)".
dataforseo.com · retrieved
“User agent string: Mozilla/5.0 (compatible; DataForSeoBot; +https://dataforseo.com/dataforseo-bot)”
DataForSEO states that DataForSeoBot obeys robots.txt and obeys crawl delay, and documents a default visit frequency of 5 seconds that a Crawl-delay directive can change.
dataforseo.com · retrieved
“Obeys robots.txt: YES Obeys crawl delay: YES”
DataForSEO documents a reverse DNS mask of crawling-gateway-*.dataforseo.com, with crawling-gateway-136-243-228-176.dataforseo.com as its example, and publishes the IPv4 and IPv6 subnets the crawler uses.
dataforseo.com · retrieved
“You can verify if a web crawler accessing your website is a legitimate DataForSEO Bot by running a reverse DNS lookup.”
Want whatever it takes to win the answer?
7 days free · card required