Bots
MJ12bot: Majestic's link crawler
MJ12bot is Majestic's link crawler. It maps links between websites for Majestic's SEO index, crawled by volunteers on their own connections, so it has no fixed IP range. It obeys robots.txt and crawl-delay up to 20 seconds. It feeds no AI answer engine, so blocking it costs no AI visibility, only your presence in Majestic's link data.
7 days free · card required
0AI-view requests from MJ12bot · 30 days
filed as conventional traffic (SEO tools, uptime checks, link unfurlers)
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| MJ12bot (this page) | not counted | Wildcard |
| AhrefsBot | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| DataForSeoBot | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| rogerbot | 0 | Wildcard |
| Screaming Frog SEO Spider | not counted | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP; it files MJ12bot as conventional traffic, off its AI view · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block MJ12bot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves MJ12bot today.
# Block Majestic's link crawler entirely.User-agent: MJ12botDisallow: / # Or keep it and slow it down. Majestic honours up to 20# seconds. Copy any wildcard-group directives you still# want into this group: Majestic states its parser prefers# the specific group over the wildcard.# User-agent: MJ12bot# Crawl-Delay: 10# No group names MJ12bot, so it reads the wildcard group.User-agent: *Allow: /Disallow: /api/We do not name MJ12bot. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs MJ12bot, and what is it for?
Majestic runs it, and we file it under product operations. Maps links between websites for Majestic's public link index, which SEO teams query. It feeds no search or AI answer engine.
From Majestic’s documentation, read .
Why does a distributed crawler change the rules?
MJ12bot does the same job as AhrefsBot, SemrushBot and Dotbot: it builds a commercial link index. Its architecture is what sets it apart. Majestic runs it as a community project, with a downloadable crawler that lets people contribute capacity from their own broadband connections, and gives its IP range as "distributed, worldwide".
The addresses look wrong. Requests arrive from scattered residential connections rather than a datacentre block. That pattern normally means a scraper or a botnet. Here it is the documented design.
You cannot block it by address. Majestic asks you not to try, because there are no consecutive IP blocks to enumerate. A firewall rule aimed at this bot will catch some of it and miss the rest, which is why the rule belongs in robots.txt. Write any Crawl-Delay into the MJ12bot group itself, because Majestic takes that group instead of the wildcard, not in addition to it.7
Why does this page state no MJ12bot version?
Majestic's page lists crawler versions with dates going back to 2017 and labels one of them current. This page states no version and no full user-agent header, because the only source for either is a page that has visibly not been maintained on that point. Match the bare token MJ12bot, case-insensitively; it is stable, and it is what the robots.txt group has to name anyway.
How do I verify that a visit really came from MJ12bot?
Not by address. Majestic publishes no range file and no reverse DNS suffix, and says a community project cannot restrict its bots to a limited set of IPs. Instead it offers a pre-arranged ident string, agreed by emailing bot@majestic12.co.uk and sent with every request to your site in a CRAWLER-IDENT header or inside the user-agent. The string is not shared with anyone else or sent to any other domain, so a request carrying it can be attributed to Majestic. It is a per-site shared secret rather than a public list, and few sites set it up. Without it, treat MJ12bot traffic as unverified. The general method is in verifying a crawler.
Majestic publishes no reverse-DNS suffix and no IP range file for MJ12bot, so a visit can be matched only by its user agent.
ResearcherTranscribed from Majestic documentation · What user-agent string does MJ12bot send?
Verbatim from Majestic’s documentation, read .
What does allowing MJ12bot cost?
Allowing it costs crawl load, which you can slow with a crawl-delay of up to 20 seconds, and keeps your links in Majestic's index. Blocking it costs no AI visibility, only your presence in Majestic's link data, which some SEO teams use to judge a site.
MJ12bot (robots.txt product token; Majestic publishes no full User-Agent header on its bot page)Should I allow MJ12bot in my situation?
Concrete cases, and what each one implies for this agent.
MJ12bot traffic is arriving from dozens of unrelated residential addresses and it looks like a botnet.
Treat the address spread as expected, and decide on the token rather than on the addresses.
Majestic runs MJ12bot as a distributed community project in which volunteers crawl from their own broadband connections. The scattered origins are the documented architecture, not evidence of spoofing — which also means an IP-based block will not work and an IP-based verification cannot work.
You want MJ12bot gone and you have already blocked it in a firewall by address.
Put the rule in robots.txt instead, and make sure robots.txt itself is reachable.
Majestic asks that the bot not be blocked by IP because it uses no consecutive IP blocks, and warns that if MJ12bot cannot retrieve robots.txt it will assume crawling is permitted. A firewall rule that also breaks the robots.txt fetch is the worst of both configurations.
You need to be certain, for a compliance log, that particular requests came from Majestic.
Email bot@majestic12.co.uk and arrange an ident string, or accept that the traffic is unverifiable.
Majestic offers a pre-arranged ident string in a CRAWLER-IDENT header or in the user-agent, which it states is not shared with anyone else or sent to any other domain. It is the only verification this operator offers, it is opt-in per site, and without it a request claiming to be MJ12bot cannot be confirmed.
What else do people ask?
Does MJ12bot obey robots.txt?
How do I slow MJ12bot down?
Does a specific MJ12bot group override my wildcard group?
Where do these facts come from?
7 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Majestic describes itself as a UK-based specialist search engine used to paint a map of the internet independent of the consumer-based search engines, and states that MJ12bot maps the link relationships between websites rather than caching web content or personal data.
mj12bot.com · retrieved
“MJ12Bot does not currently cache web content or personal data. Instead it maps the link relationships between websites to build a search engine.”
Majestic documents MJ12bot as the robots.txt user-agent token for its crawler, and publishes no full User-Agent header string for it on its bot page.
mj12bot.com · retrieved
“User-agent: MJ12bot”
Majestic states that MJ12bot obeys robots.txt and obeys crawl delay, and that with a Crawl-Delay set it will make up to a 20-second delay between requests to a site.
mj12bot.com · retrieved
“Obeys Robots.txt Yes”
Majestic asks that MJ12bot not be blocked by IP because it does not use consecutive IP blocks, and warns that if the bot cannot retrieve robots.txt it will assume that it is okay to crawl the site.
mj12bot.com · retrieved
“Please always make sure the bot can actually retrieve robots.txt itself. If it can't then it will assume that it is okay to crawl your site.”
Majestic states that MJ12bot is a distributed crawler run as a community project, enabling people with broadband connections to contribute crawling capacity, and describes its IP range as distributed and worldwide.
mj12bot.com · retrieved
“a fast and efficient downloadable distributed crawler that enables people with broadband connections to help contribute”
Majestic states that as a community project it cannot restrict its bots to a limited number of IP addresses, and instead offers a pre-arranged ident string sent in the CRAWLER-IDENT header or as part of the user-agent, arranged by contacting bot@majestic12.co.uk.
mj12bot.com · retrieved
“However we can send a pre-arranged ident string with all requests to your site. This can be sent as part of the http or https headers in the 'CRAWLER-IDENT' field, or as part of the User-Agent string.”
Majestic states that if a robots.txt file has an MJ12bot section, that section will be taken in preference over the wildcard section rather than in addition to it, so a crawl-delay in the wildcard section must be copied into the MJ12bot section.
mj12bot.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required