Bots
SemrushBot: Semrush's ten crawlers
SemrushBot is Semrush's crawler family: ten robots.txt tokens, one per Semrush tool, from the backlink index to site audits. A rule naming SemrushBot leaves the other nine untouched. It feeds no AI answer engine, so blocking costs no AI visibility. Semrush publishes no IP ranges, so a visit cannot be verified. attensira.com blocks SemrushBot.
7 days free · card required
0AI-view requests from SemrushBot · 30 days
filed as conventional traffic (SEO tools, uptime checks, link unfurlers)
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| SemrushBot (this page) | not counted | Blocked |
| AhrefsBot | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| DataForSeoBot | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| rogerbot | 0 | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP; it files SemrushBot as conventional traffic, off its AI view · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block SemrushBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves SemrushBot today.
# Semrush uses one token per tool. Blocking the family# takes ten groups.User-agent: SemrushBotDisallow: / User-agent: SiteAuditBotDisallow: / User-agent: SemrushBot-BADisallow: / User-agent: SemrushBot-SIDisallow: / User-agent: SemrushBot-SWADisallow: / User-agent: SplitSignalBotDisallow: / User-agent: SemrushBot-OCOBDisallow: / User-agent: SemrushBot-FTDisallow: / User-agent: RyteBotDisallow: / User-agent: SemrushBot-ESIDisallow: /User-agent: SemrushBotDisallow: /We name SemrushBot and disallow the whole site. Nothing else in the file changes for it.
Read the whole file, or write yours with the robots.txt generator.
Who runs SemrushBot, and what is it for?
Semrush runs it, and we file it under product operations. Feeds Semrush's tools, from the backlink index to site audits, depending on which of its ten tokens is calling. It feeds no AI answer engine.
From Semrush’s documentation, read .
Why does a SemrushBot block leave Semrush traffic in my logs?
robots.txt matches by token, and none of the nine sibling tokens is SemrushBot. A group naming SemrushBot reaches the backlink crawler and nothing else, so "I blocked Semrush and they are still crawling me" is usually a rule that never addressed the agent that showed up, not a compliance failure.3
The fix is to name each token you mean. Reaching for User-agent: * instead also shuts out the search engines and the AI crawlers that can cite you. Our robots.txt generator assembles the groups if you would rather not maintain ten by hand.
How long does a robots.txt change take to reach SemrushBot?
Semrush says it may take up to one hour or 100 requests for SemrushBot to discover a change.4 Judge a new rule the next day, not in the next ten minutes. The same page says the backlink crawler honours Crawl-delay up to 10 seconds and cuts higher values down to that limit, so a delay of 60 behaves like a delay of 10.
How do I verify that a visit really came from SemrushBot?
You cannot. Semrush says "Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks", and publishes no range file and no reverse DNS suffix.5 Where AhrefsBot or DataForSeoBot log lines can be checked with one lookup, a SemrushBot line stays a claim. It is also a name worth borrowing, because many sites allowlist it at the CDN. So build no allowlist exception around it, and when something claiming to be SemrushBot misbehaves, respond to the behaviour rather than the name. The general reasoning is in verifying a crawler.
Semrush publishes no reverse-DNS suffix and no IP range file for SemrushBot, so a visit can be matched only by its user agent.
ResearcherTranscribed from Semrush documentation · What user-agent string does SemrushBot send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
What does allowing SemrushBot cost?
Allowing it costs crawl load across up to ten tokens and keeps your site in Semrush's data. Blocking costs no AI visibility, but a rule naming SemrushBot leaves the other nine tokens untouched, so a real block names each one you mean.
SemrushBot (the robots.txt product token Semrush documents)Should I allow SemrushBot in my situation?
Concrete cases, and what each one implies for this agent.
You blocked SemrushBot months ago and Semrush traffic is still in your logs.
Add the nine sibling tokens, not a wildcard.
Semrush documents ten tokens and gives ten separate snippets. SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, SemrushBot-OCOB, SemrushBot-FT, RyteBot and SemrushBot-ESI are each addressed by name, and none of them is covered by a group that says SemrushBot. Reaching for User-agent -asterisk- instead would also catch the search and AI crawlers you probably want.
A crawl claiming to be SemrushBot is hitting login forms and URLs that never existed.
Rate-limit or block at the edge on behaviour, not on the name.
Semrush publishes no IP list and no reverse DNS suffix, so you cannot confirm the request is theirs. When an operator gives you no verification method, the identity in the header stops being usable evidence and the only reliable signal left is what the client is actually doing.
The crawl is welcome but the pace is not.
Set Crawl-delay for SemrushBot rather than disallowing it.
Semrush states that SemrushBot for Backlink Analytics supports crawl-delay, that intervals of up to 10 seconds are honoured, and that higher values are cut down to that limit. It also notes it may take up to an hour or 100 requests for a robots.txt change to be picked up, so do not judge the fix in the first ten minutes.
What else do people ask?
Can I keep Semrush Site Audit on my own site and still block the backlink crawler?
Which Semrush tools are behind RyteBot and SemrushBot-ESI?
Why does attensira.com block SemrushBot?
Where do these facts come from?
5 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Semrush describes SemrushBot as the search bot software that Semrush sends out to discover and collect new and updated web data, feeding tools including Backlink Analytics, Site Audit, Backlink Audit and Link Building.
semrush.com · retrieved
“SemrushBot is the search bot software that Semrush sends out to discover and collect new and updated web data.”
Semrush documents SemrushBot as the robots.txt user-agent token for its backlink webgraph crawler.
semrush.com · retrieved
“User-agent: SemrushBot”
Semrush documents ten robots.txt tokens — SemrushBot, SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, SemrushBot-OCOB, SemrushBot-FT, RyteBot and SemrushBot-ESI — each with its own Disallow snippet for a different Semrush tool.
semrush.com · retrieved
Semrush states that SemrushBot for Backlink Analytics supports Crawl-delay directives with intervals of up to 10 seconds and wildcards, and that it may take up to one hour or 100 requests for SemrushBot to discover changes to robots.txt.
semrush.com · retrieved
“Our crawler can take intervals of up to 10 seconds between requests to a site.”
Semrush states that it does not use consecutive IP blocks and advises against blocking SemrushBot by IP; the page publishes no IP range list and no reverse DNS suffix.
semrush.com · retrieved
“Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks.”
Want whatever it takes to win the answer?
7 days free · card required