Ever wonder which SEO bots are being blocked the most—and what that means for your SEO insights?
We analyzed data from over 140 million websites and consulted with data scientist Xibeijia Guan to find out which crawlers are getting turned away most frequently. The results might surprise you—and they could be impacting the quality of your SEO tools’ link data more than you think.
🔍 Why Blocking SEO Bots Matters
Blocking SEO bots can significantly affect the backlink index and link intelligence of SEO platforms. When a bot can’t access your site:
- It can’t index outbound or internal links, which affects link profiles.
- Tools like Ahrefs won’t show accurate page history or internal link structures.
- Your site might not appear in alternative search engines, like Yep.com (powered by AhrefsBot).
However, traffic estimates, keyword rankings, and top page data are typically unaffected. Those metrics are generated from other data sources, like clickstream data or browser plugins.
Top 3 Most Blocked SEO Bots (By Website Percentage)
Based on robots.txt directives from millions of sites, the top blocked SEO bots are:
| Bot Name | Block Rate | Operator |
|---|---|---|
| MJ12bot | 6.49% | Majestic |
| SemrushBot | 6.34% | Semrush |
| AhrefsBot | 6.31% | Ahrefs |
These block rates indicate a growing trend in webmasters pushing back against high-frequency crawlers—especially ones from link intelligence platforms.
Understanding the Blocking Methods
Websites can block bots through various robots.txt rules, including:
User-agent: *withDisallow: /(blocks all bots)- Specific bot disallow rules (e.g.,
User-agent: AhrefsBot) - Allow overrides after a general block
🔒 Note: This data only considers robots.txt directives—not firewall, server-level, or IP-based blocks.
Why MJ12bot Leads in Block
MJ12bot, from Majestic, is the most blocked for several reasons:
- It’s a distributed crawler—difficult to identify or block by IP.
- Has been around longer than many competitors.
- Lacks a broad user base, so site owners have less incentive to allow access.
Detailed Breakdown: Blocked SEO Bots (By Count)
| Bot Name | Blocked Sites | % of Sample | Operator |
|---|---|---|---|
| MJ12bot | 9,081,205 | 6.49% | Majestic |
| SemrushBot | 8,868,486 | 6.34% | Semrush |
| AhrefsBot | 8,831,316 | 6.31% | Ahrefs |
| dotbot | 8,569,766 | 6.13% | Moz |
| BLEXBot | 8,374,216 | 5.99% | SEO PowerSuite |
| serpstatbot | 7,878,935 | 5.63% | Serpstat |
| DataForSeoBot | 7,872,939 | 5.63% | DataForSEO |
| SemrushBot-CT | 7,855,400 | 5.62% | Semrush |
| Barkrowler | 7,804,425 | 5.58% | Babbar |
| SemrushBot-BA | 7,796,785 | 5.57% | Semrush |
Explicit Bot Blocking: Who’s Getting Targeted?
When site owners specifically name and disallow a bot, that’s an explicit block—not just a general one. Here’s who gets hit the most:
| Bot Name | Explicit Blocks | % of Sites | Operator |
|---|---|---|---|
| MJ12bot | 2,000,372 | 1.43% | Majestic |
| dotbot | 1,402,305 | 1.00% | Moz |
| AhrefsBot | 1,350,771 | 0.97% | Ahrefs |
| SemrushBot | 1,285,857 | 0.92% | Semrush |
| BLEXBot | 861,184 | 0.62% | SEO PowerSuite |
These bots are seen as over-aggressive, or simply less trusted by site owners.
By the Numbers: High DR Sites Blocking SEO Bots
Among the top 1 million domains (by Domain Rating > 45), SemrushBot tops the list in total blocks—indicating higher scrutiny from more authoritative websites.
SEO Bot Blocking by Industry
Certain industries block SEO bots more than others. The top three categories are:
- Automotive (39%)
- Books & Literature (27%)
- Real Estate (17%)
These sectors may prioritize content privacy, crawler fatigue, or technical SEO control.
Fastest SEO Crawlers (According to Cloudflare Radar)
When it comes to speed, AhrefsBot is the fastest SEO bot in the game:
- 4.6x faster than Moz’s Dotbot
- 6.7x faster than SemrushBot
While fast crawling means more frequent updates, it may also increase the chance of being blocked if the crawl load feels excessive.
Final Thoughts: Should You Block SEO Bots?
Blocking bots can preserve bandwidth and privacy, but it comes at a cost:
- You may disappear from tools your competitors use to find backlink opportunities.
- Features like Ahrefs’ link audits or Majestic’s link graphs may no longer reflect your site accurately.
- Blocking crawlers from reputable SEO tools could limit your discoverability on alternative engines like Yep.com.
Recommendation: Instead of a blanket block, use crawl-delay, rate-limiting, or segment-specific robots.txt to manage bot access smartly.
