--- title: "SEO and backlink crawlers (17) — AI Crawler Index" description: "Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth." canonical: "https://www.pathwren.workers.dev/category/seo.html" url: "https://www.pathwren.workers.dev/category/seo.md" format: "markdown" source: "the bytes of /category/seo.html, in the build that wrote the page" generator: "surfaces/ai-crawler-index/build.py" generated: "2026-09-03T06:29:54+00:00" license: "CC0-1.0" --- # SEO and backlink crawlers > Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth. Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth. | Crawler | Token | Operator | Cost of blocking | | --- | --- | --- | --- | | [AhrefsBot](https://www.pathwren.workers.dev/crawler/ahrefsbot.html) | `AhrefsBot` | Ahrefs | No user-facing effect. Ahrefs honours Crawl-delay, so rate-limiting is usually better than… | | [AhrefsSiteAudit](https://www.pathwren.workers.dev/crawler/ahrefssiteaudit.html) | `AhrefsSiteAudit` | Ahrefs | Site owners auditing your domain get an incomplete report. Blocking it saves bandwidth and… | | [Barkrowler](https://www.pathwren.workers.dev/crawler/barkrowler.html) | `barkrowler` | Babbar | You leave Babbar's index. No effect on search or assistants.… | | [DataForSeoBot](https://www.pathwren.workers.dev/crawler/dataforseobot.html) | `DataForSeoBot` | DataForSEO | You leave a dataset that a long tail of SEO products is built on. No user-facing effect.… | | [DotBot](https://www.pathwren.workers.dev/crawler/dotbot.html) | `dotbot` | Moz | You leave Moz's link index, so Domain Authority and link reports about your site get thinn… | | [MJ12bot](https://www.pathwren.workers.dev/crawler/mj12bot.html) | `MJ12bot` | Majestic | You leave the Majestic backlink index. No search or AI effect. It supports Crawl-delay, wh… | | [rogerbot](https://www.pathwren.workers.dev/crawler/rogerbot.html) | `rogerbot` | Moz | Moz Pro site audits of your domain stop. If the domain is yours and you use Moz, blocking … | | [SemrushBot](https://www.pathwren.workers.dev/crawler/semrushbot.html) | `SemrushBot` | Semrush | Your competitors' Semrush reports get thinner, and so do yours. No user-facing effect.… | | [SemrushBot-BA](https://www.pathwren.workers.dev/crawler/semrushbot-ba.html) | `SemrushBot-BA` | Semrush | None to you. It costs the site being audited a little accuracy in their backlink report.… | | [SemrushBot-ESI](https://www.pathwren.workers.dev/crawler/semrushbot-esi.html) | `SemrushBot-ESI` | Semrush | Enterprise customers lose analysis of your domain. Nothing user-facing.… | | [SemrushBot-FT](https://www.pathwren.workers.dev/crawler/semrushbot-ft.html) | `SemrushBot-FT` | Semrush | Your text stops being compared against other people's submissions — which also means copie… | | [SemrushBot-SI](https://www.pathwren.workers.dev/crawler/semrushbot-si.html) | `SemrushBot-SI` | Semrush | Nothing user-facing. Semrush customers lose on-page suggestions for pages on your domain.… | | [SemrushBot-SWA](https://www.pathwren.workers.dev/crawler/semrushbot-swa.html) | `SemrushBot-SWA` | Semrush | Writers using Semrush's assistant see your links reported as unreachable.… | | [SEOkicks](https://www.pathwren.workers.dev/crawler/seokicks.html) | `SEOkicks` | SEOkicks | You leave a regional backlink index. Nothing else changes.… | | [serpstatbot](https://www.pathwren.workers.dev/crawler/serpstatbot.html) | `serpstatbot` | Serpstat | You leave Serpstat's link index. Try Crawl-delay first — they honour it, and a slow crawle… | | [SiteAuditBot](https://www.pathwren.workers.dev/crawler/siteauditbot.html) | `SiteAuditBot` | Semrush | Site Audit reports on your own domain stop working. If a customer is auditing your site wi… | | [SplitSignalBot](https://www.pathwren.workers.dev/crawler/splitsignalbot.html) | `SplitSignalBot` | Semrush | A site owner's own A/B testing stops. Only relevant on domains whose owner uses the produc… | [json](https://www.pathwren.workers.dev/category/seo.json) ## Sitemap - [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates - [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable - [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file - [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag - [A2A agents](https://www.pathwren.workers.dev/a2a.html) - [About and method](https://www.pathwren.workers.dev/about.html) - [API](https://www.pathwren.workers.dev/api.html) - [Changelog](https://www.pathwren.workers.dev/changelog.html) - [Compliance](https://www.pathwren.workers.dev/compliance) - [Contact](https://www.pathwren.workers.dev/contact) - [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum) - [AI Crawler Index](https://www.pathwren.workers.dev/index.html) - [No model runs here](https://www.pathwren.workers.dev/inference.html) - [Legal](https://www.pathwren.workers.dev/legal) - [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html) - [MCP server](https://www.pathwren.workers.dev/mcp-lint.html) - [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html) - [MCP server](https://www.pathwren.workers.dev/mcp-robots.html) - [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html) - [MCP server](https://www.pathwren.workers.dev/mcp-triage.html) - [MCP server](https://www.pathwren.workers.dev/mcp.html) - [Packages](https://www.pathwren.workers.dev/packages.html) - [Pricing](https://www.pathwren.workers.dev/pricing) - [Privacy](https://www.pathwren.workers.dev/privacy.html) - [API reference](https://www.pathwren.workers.dev/reference) - [Access, keys and sign-up](https://www.pathwren.workers.dev/register) - [Security posture](https://www.pathwren.workers.dev/security.html) - [Upstream status](https://www.pathwren.workers.dev/status.html) - [Terms of use](https://www.pathwren.workers.dev/terms.html) - [Trust](https://www.pathwren.workers.dev/trust) ## Machine copies of this page - [HTML (canonical)](https://www.pathwren.workers.dev/category/seo.html) - [JSON](https://www.pathwren.workers.dev/category/seo.json) - [Markdown](https://www.pathwren.workers.dev/category/seo.md) — this document This document is a markdown rendering of [https://www.pathwren.workers.dev/category/seo.html](https://www.pathwren.workers.dev/category/seo.html), generated from that page's own bytes in the same build. The HTML page is canonical.