Bots
AhrefsSiteAudit: allow it or block it?
AhrefsSiteAudit is the crawler behind Ahrefs Site Audit: it crawls a site when someone with an Ahrefs account audits it, checking for problems such as broken links and slow pages. Allow it, especially if your team uses Ahrefs. Blocking it does not keep you out of the Ahrefs link database, which is AhrefsBot's job; it only makes audits incomplete.
7 days free · card required
0requests filed as AhrefsSiteAudit · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| AhrefsSiteAudit (this page) | 0 | Wildcard |
| AhrefsBot | not counted | Blocked |
| DataForSeoBot | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| rogerbot | 0 | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block AhrefsSiteAudit in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves AhrefsSiteAudit today.
# Let the audit crawler in, keep the link-database crawler# out.User-agent: AhrefsSiteAuditAllow: / User-agent: AhrefsBotDisallow: /# No group names AhrefsSiteAudit, so it reads the wildcard# group.User-agent: *Allow: /Disallow: /api/We do not name AhrefsSiteAudit. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs AhrefsSiteAudit, and what is it for?
Ahrefs runs it, and we file it under product operations. Crawls for an SEO tool: a link index, a site audit or a test fetch. It feeds no AI answer and, apart from AhrefsBot's work for Yep, no search engine's index.
From Ahrefs’s documentation, read .
Who sent AhrefsSiteAudit to your site?
Someone with an Ahrefs account, and possibly you. Ahrefs runs two crawlers. AhrefsSiteAudit is documented as powering the Site Audit tool, which checks websites for technical and on-page issues such as broken links and slow pages1. AhrefsBot is the other: it crawls the web on Ahrefs' own initiative to build a database any subscriber can query about any site.
Both fetch ordinary pages. What separates them is who asked. AhrefsSiteAudit crawls a site because someone pointed it there, and the results go back to that person. If that person is on your team, blocking it is self-defeating in a way that blocking AhrefsBot is not.
The robots.txt block above is the split for a site that uses Ahrefs audits but objects to the link database: audit crawler allowed, database crawler disallowed.
Why is your audit missing pages?
Usually because the crawler did what robots.txt told it. Ahrefs states that AhrefsSiteAudit obeys robots.txt by default, and that website owners can request the crawler disobey robots.txt on their own site, so they can check sections normally disallowed for crawlers4.
That exception is the honest version of something several tools do quietly. An audit crawler that respects your Disallow reports nothing about the disallowed sections. Not "blocked"; often just fewer pages. A report that silently omits a third of the site reads exactly like a clean one. If the missing pages all sit under a path you blocked years ago, that is the cause, and the opt-in bypass, available only to the verified owner, is the documented fix.
It is also a reminder that "obeys robots.txt" usually has conditions attached.
How do I verify that a visit really came from AhrefsSiteAudit?
A user-agent header is free text, so a log line naming AhrefsSiteAudit is a claim until you check it.
Look up the visiting IP, confirm the hostname ends in one of the suffixes in the record, then look that hostname up and confirm it returns the same IP, or match the IP against the published ranges.
The IP lists and reverse DNS suffix cover all of Ahrefs' crawling, so a verified address proves the request came from Ahrefs, not which of its two crawlers sent it. Check the address first, then read the token to tell AhrefsSiteAudit from AhrefsBot.
Why does AhrefsSiteAudit show up as Android traffic?
Because of how its mobile header is built. Ahrefs documents a short desktop string, Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; …) ending in its site-audit help URL2, and a mobile string that is a full Android Chrome header with the same token added in the compatible clause3.
A log filter that classifies traffic by the front of the user-agent string files those mobile requests as a person on an Android phone. That is how a bot report in an analytics dashboard ends up counting a crawler as a visitor, and it is not unique to Ahrefs: every crawler that emulates a mobile browser produces the same effect. Match on the token, wherever in the string it sits. Both strings carry versions Ahrefs increments, so match the bare AhrefsSiteAudit.
What does the 30-URL-a-minute cap tell you?
Ahrefs states the bot limits crawling to 30 URLs per minute at most, to prevent excessive load5. The cap is useful beyond reassurance: it is a threshold. Traffic claiming to be AhrefsSiteAudit at much more than one request every two seconds is either not Ahrefs or not behaving as documented. Either way, the next step is verification.
It also sets expectations for your own audits. At 30 URLs a minute the crawler fetches at most 1,800 URLs an hour, so a 10,000-URL site takes at least five and a half hours to audit, and a 100,000-URL site at least 55. An audit that finishes much faster than that did not crawl the whole site, which is worth knowing before you read its report as complete.
- Reverse DNS ends in
- *.ahrefs.com*.ahrefs.net
- IP range files
- crawler-ip-rangescrawler-ips
ResearcherTranscribed from Ahrefs documentation · Agents filed beside AhrefsSiteAudit
What user-agent string does AhrefsSiteAudit send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
The version in this string changes; check Ahrefs’s page before relying on the exact text.
How does it compare with other audit crawlers?
AhrefsSiteAudit
- Who starts the crawl
- An Ahrefs user auditing a site
- What it feeds
- That user's Site Audit report
AhrefsBot
- Who starts the crawl
- Ahrefs, continuously
- What it feeds
- The Ahrefs link database and Yep
- Who starts the crawl
- Whoever runs the desktop app
- What it feeds
- That person's crawl
None of these is an AI crawler, and none affects whether AI engines cite you.
What can Attensira show you about AhrefsSiteAudit?
AI traffic, on Growth and Business, groups crawler fetches by operator and declared purpose at the edge, so audit crawls do not inflate the numbers you read for AI crawling. It reports user agents as sent. See crawler logs and the robots.txt generator.
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http:/Should I allow AhrefsSiteAudit in my situation?
Concrete cases, and what each one implies for this agent.
Your Ahrefs site audit reports far fewer pages than the site has, and the missing ones are all under a path you blocked years ago.
Either allow AhrefsSiteAudit on that path, or ask Ahrefs to run the audit ignoring robots.txt.
Ahrefs states that the crawler obeys robots.txt by default and that website owners can request it disobey robots.txt on their own site precisely so they can check sections that are normally disallowed. An audit that silently skips a third of the site is worse than no audit, because it reads as a clean bill of health.
You want out of the Ahrefs link database but your SEO lead runs weekly audits with the same account.
Disallow AhrefsBot and allow AhrefsSiteAudit, as two separate robots.txt groups.
The two tokens have two documented jobs. AhrefsBot builds the shared database that anyone with a subscription can query about your site. AhrefsSiteAudit crawls your site because someone asked it to and returns the result to them. Blocking one and allowing the other is coherent, and it is invisible if you write a single rule against a wildcard instead.
A burst of AhrefsSiteAudit traffic is arriving much faster than you expected and you suspect it is not really Ahrefs.
Check the rate against the documented cap, then verify the addresses.
Ahrefs states the crawler is limited to 30 URLs per minute. Traffic claiming the token at a materially higher rate is either not Ahrefs or not behaving as documented, and in both cases the address check against the published ranges settles it in one step.
What else do people ask?
Why does my Ahrefs site audit show fewer pages than my site has?
How fast does AhrefsSiteAudit crawl?
Why does AhrefsSiteAudit look like a phone in my analytics?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Ahrefs documents AhrefsSiteAudit as the crawler that powers its Site Audit tool, which checks websites for technical and on-page issues such as broken links and slow pages.
ahrefs.com · retrieved
“AhrefsSiteAudit powers our Site Audit tool.”
Ahrefs documents the desktop user-agent string of AhrefsSiteAudit as "Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)".
ahrefs.com · retrieved · changes often, check the source
Ahrefs documents a second, mobile user-agent string for AhrefsSiteAudit that wraps the same AhrefsSiteAudit token inside an Android Chrome header — "Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.173 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)".
ahrefs.com · retrieved · changes often, check the source
Ahrefs states that AhrefsSiteAudit obeys robots.txt by default, and that website owners can request the crawler disobey robots.txt on their own site so they can check for issues in sections normally disallowed for crawlers.
ahrefs.com · retrieved
“Obeys robots.txt: Yes by default (website owners can request to disobey robots.txt on their sites)”
Ahrefs states that the AhrefsSiteAudit bot prevents excessive load on website servers by limiting crawling to 30 URLs per minute maximum.
ahrefs.com · retrieved
“limiting crawling to 30 URLs per min max”
Ahrefs publishes crawler IP ranges at api.ahrefs.com/v3/public/crawler-ip-ranges and individual IPs at api.ahrefs.com/v3/public/crawler-ips, and states the reverse DNS suffix of the IP's hostname is always ahrefs.com or ahrefs.net.
ahrefs.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required