Bots
What is AhrefsBot, and should you allow it?
AhrefsBot is the crawler that builds the Ahrefs link database and the Yep search engine's index. It is not an AI crawler: allowing or blocking it changes what Ahrefs subscribers can see about your backlinks, not what AI answers say about you. Block it to keep your link data out; slow it with Crawl-Delay if load is the issue.
7 days free · card required
0AI-view requests from AhrefsBot · 30 days
filed as conventional traffic (SEO tools, uptime checks, link unfurlers)
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| AhrefsBot (this page) | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| DataForSeoBot | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| rogerbot | 0 | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP; it files AhrefsBot as conventional traffic, off its AI view · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block AhrefsBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves AhrefsBot today.
# Keep Ahrefs' link crawler out of the whole site.User-agent: AhrefsBotDisallow: / # Or slow it down instead of blocking it.# User-agent: AhrefsBot# Crawl-Delay: 10User-agent: AhrefsBotDisallow: /We name AhrefsBot and disallow the whole site. Nothing else in the file changes for it.
Read the whole file, or write yours with the robots.txt generator.
Who runs AhrefsBot, and what is it for?
Ahrefs runs it, and we file it under product operations. Builds the Ahrefs link database and the index of Yep, Ahrefs' own search engine. Ahrefs describes no AI assistant it feeds, so a rule here decides what Ahrefs subscribers and Yep see, not what AI answers say.
From Ahrefs’s documentation, read .
Does AhrefsBot affect your AI visibility?
No. AhrefsBot is on a different axis entirely. Ahrefs documents it as powering the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine1. Nothing it collects is described as going into a model's training corpus, and nothing it collects puts your URL next to an answer in an assistant.
So the decision is commercial. Allowing it keeps your backlink profile visible to every Ahrefs subscriber, including your competitors. Blocking it removes you from that dataset. The Yep index means AhrefsBot could in principle send a human to your site; in practice the decision people make is about the database. If you came here from a question about ChatGPT or AI Overviews, this crawler does not change the answer.
Why does attensira.com block AhrefsBot?
Because we would rather state our position than pretend to neutrality. attensira.com's robots.txt disallows AhrefsBot, SemrushBot and CCBot. The comment in our source code gives the reason for the first two plainly: they crawl us to sell our data back to competitors.
That is a commercial preference, not a recommendation. Everywhere else in this directory we argue for allowing the crawlers that can cite you and being deliberate about the ones that only train on you. AhrefsBot is neither. Blocking it is a choice about who may package our link profile as a product. Many publishers reasonably choose the other way, because being visible in the tools their partners and agencies use is worth more to them than the data.
Should you block AhrefsBot or just slow it down?
It depends on what the objection is. Ahrefs states that AhrefsBot obeys robots.txt and obeys crawl delay3. That turns a binary into a dial.
- The objection is the data. Block it. The robots.txt block above disallows the whole site.
- The objection is load. The site stays up but response times climb when Ahrefs crawls. Use the commented
Crawl-Delayvariant in the same block instead. A full block is an overreaction that also hides your site from your own team's research in the tool.
Write the rule against the right token. AhrefsSiteAudit is a separate crawler that only crawls a site when someone with an Ahrefs account audits it5. A group naming AhrefsBot leaves it alone, which is how you keep out of the database and still audit your own site.
How do I verify that a visit really came from AhrefsBot?
A user-agent header is free text, so a log line naming AhrefsBot is a claim until you check it.
Look up the visiting IP, confirm the hostname ends in one of the suffixes in the record, then look that hostname up and confirm it returns the same IP, or match the IP against the published ranges.
Ahrefs publishes a CIDR range list, an individual-address list and a reverse DNS suffix (ahrefs.com or ahrefs.net), so you can check one against the other: a range file can be stale between fetches, a reverse lookup is live. Bother, because AhrefsBot is a token many sites allowlist at the CDN, which makes it worth borrowing.
How is AhrefsBot different from SemrushBot?
They do the same kind of job for two competing platforms. Semrush describes SemrushBot as the software it sends out to discover and collect web data, feeding tools including Backlink Analytics, Site Audit, Backlink Audit and Link Building6. The differences are in how much each operator lets you check and control.
Feeds
- AhrefsBot (Ahrefs)
- Ahrefs link database and Yep search
- SemrushBot (Semrush)
- Backlink Analytics and other Semrush tools
Robots.txt tokens
- AhrefsBot (Ahrefs)
- AhrefsBot, plus AhrefsSiteAudit for audits
- SemrushBot (Semrush)
- Ten tokens, one per tool7
Crawl-delay
- AhrefsBot (Ahrefs)
- Obeyed
- SemrushBot (Semrush)
- Supported up to 10 seconds; robots.txt changes may take up to an hour or 100 requests to be seen9
Verification
- AhrefsBot (Ahrefs)
- IP range list, address list and reverse DNS
- SemrushBot (Semrush)
- No IP list, no reverse DNS suffix; Semrush advises against IP blocking8
The token count is where Semrush rules go wrong. A group naming only SemrushBot covers the backlink crawler and leaves other Semrush tools, such as its own site audit bot, untouched. With Ahrefs, two tokens cover the whole operator. And because Semrush publishes no way to verify its crawler, a SemrushBot count in your logs is a count of claims. SemrushBot has its own page.
- Reverse DNS ends in
- *.ahrefs.com*.ahrefs.net
- IP range files
- crawler-ip-rangescrawler-ips
ResearcherTranscribed from Ahrefs documentation · Agents filed beside AhrefsBot
How should you match AhrefsBot in logs?
Ahrefs documents the header as AhrefsBot/7.02. The 7.0 is a version Ahrefs increments. Match the bare token AhrefsBot, case-insensitively; a rule pinned to AhrefsBot/7.0 stops matching on the day version 8 ships.
What can Attensira show you about AhrefsBot?
Mostly that it is not part of your AI picture. AI traffic, on Growth and Business, groups crawler fetches by operator and declared purpose at the edge, so SEO-tool crawlers sit apart from the AI agents that affect answers. It reports user agents as sent. What AI engines say about you is read in the answers themselves, every day. See crawler logs and the robots.txt generator.
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http:/Should I allow AhrefsBot in my situation?
Concrete cases, and what each one implies for this agent.
Your bandwidth bill jumped and the logs are full of a token you do not recognise.
Verify the addresses against Ahrefs' published ranges before you write any rule.
Ahrefs publishes its crawler addresses at two endpoints and states that the reverse DNS hostname always ends in ahrefs.com or ahrefs.net. If the addresses do not match, you are looking at a scraper wearing the name of a bot that obeys robots.txt — and the robots.txt rule you were about to write would have had no effect on it.
You want to keep your backlink profile out of a competitor's research tool, but you still want to audit your own site with Ahrefs.
Disallow AhrefsBot and leave AhrefsSiteAudit allowed.
These are two separate tokens with two separate documented jobs. AhrefsBot builds the shared link database; AhrefsSiteAudit powers the Site Audit tool that a site owner runs against their own site. Blocking the first and allowing the second is a coherent position and takes two robots.txt groups.
The crawl is not the problem, the rate is — the site stays up but response times climb whenever Ahrefs runs.
Set a crawl-delay for AhrefsBot rather than disallowing it.
Ahrefs states that AhrefsBot obeys crawl-delay. That makes the rate a negotiable parameter rather than an all-or-nothing decision, and it is a strictly better answer than a block if your objection was load rather than data collection.
What else do people ask?
Is AhrefsBot an AI crawler?
Does AhrefsBot respect robots.txt and Crawl-delay?
What is the difference between AhrefsBot and SemrushBot?
Does blocking AhrefsBot stop Ahrefs Site Audit?
Where do these facts come from?
9 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Ahrefs documents AhrefsBot as powering the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.
ahrefs.com · retrieved
“Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.”
Ahrefs documents AhrefsBot's user-agent string as "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" and its robots.txt token as AhrefsBot.
ahrefs.com · retrieved · changes often, check the source
Ahrefs states that AhrefsBot obeys robots.txt and obeys crawl delay.
ahrefs.com · retrieved
“Obeys robots.txt: Yes”
Ahrefs publishes its crawler IP ranges at api.ahrefs.com/v3/public/crawler-ip-ranges and individual IPs at api.ahrefs.com/v3/public/crawler-ips, and states that the reverse DNS suffix of the IP's hostname is always ahrefs.com or ahrefs.net.
ahrefs.com · retrieved
“The reverse DNS suffix of the IPs hostname is always ahrefs.com or ahrefs.net.”
Ahrefs documents AhrefsSiteAudit as a separate crawler, with its own robots.txt token, that powers the Ahrefs Site Audit tool.
ahrefs.com · retrieved
Semrush describes SemrushBot as the search bot software Semrush sends out to discover and collect new and updated web data, feeding tools including Backlink Analytics, Site Audit, Backlink Audit and Link Building.
semrush.com · retrieved
Semrush documents ten robots.txt tokens, including SemrushBot and SiteAuditBot, each with its own Disallow snippet for a different Semrush tool.
semrush.com · retrieved
Semrush states that it does not use consecutive IP blocks and advises against blocking SemrushBot by IP; its bot page publishes no IP range list and no reverse DNS suffix.
semrush.com · retrieved
Semrush states that SemrushBot for Backlink Analytics supports Crawl-delay directives with intervals of up to 10 seconds, and that it may take up to one hour or 100 requests for SemrushBot to discover changes to robots.txt.
semrush.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required