---
title: "robots.txt: Block SEO and backlink crawlers — AI Crawler Index"
description: "Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic."
canonical: "https://www.pathwren.workers.dev/policy/block-seo-tools.html"
url: "https://www.pathwren.workers.dev/policy/block-seo-tools.md"
format: "markdown"
source: "the bytes of /policy/block-seo-tools.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# Block SEO and backlink crawlers

> Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic.

Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic.

```bash
curl -s https://www.pathwren.workers.dev/robots/block-seo-tools.txt >> robots.txt
```

The cheapest bandwidth saving available to most sites, and the one nobody regrets. The only cost is that your own dashboards on those tools get thinner. Consider Crawl-delay first: both major operators honour it.

## Names 17 crawlers

[AhrefsBot](https://www.pathwren.workers.dev/crawler/ahrefsbot.html) · [AhrefsSiteAudit](https://www.pathwren.workers.dev/crawler/ahrefssiteaudit.html) · [Barkrowler](https://www.pathwren.workers.dev/crawler/barkrowler.html) · [DataForSeoBot](https://www.pathwren.workers.dev/crawler/dataforseobot.html) · [DotBot](https://www.pathwren.workers.dev/crawler/dotbot.html) · [MJ12bot](https://www.pathwren.workers.dev/crawler/mj12bot.html) · [rogerbot](https://www.pathwren.workers.dev/crawler/rogerbot.html) · [SemrushBot](https://www.pathwren.workers.dev/crawler/semrushbot.html) · [SemrushBot-BA](https://www.pathwren.workers.dev/crawler/semrushbot-ba.html) · [SemrushBot-ESI](https://www.pathwren.workers.dev/crawler/semrushbot-esi.html) · [SemrushBot-FT](https://www.pathwren.workers.dev/crawler/semrushbot-ft.html) · [SemrushBot-SI](https://www.pathwren.workers.dev/crawler/semrushbot-si.html) · [SemrushBot-SWA](https://www.pathwren.workers.dev/crawler/semrushbot-swa.html) · [SEOkicks](https://www.pathwren.workers.dev/crawler/seokicks.html) · [serpstatbot](https://www.pathwren.workers.dev/crawler/serpstatbot.html) · [SiteAuditBot](https://www.pathwren.workers.dev/crawler/siteauditbot.html) · [SplitSignalBot](https://www.pathwren.workers.dev/crawler/splitsignalbot.html)

## The file

[/robots/block-seo-tools.txt](https://www.pathwren.workers.dev/robots/block-seo-tools.txt) · [json](https://www.pathwren.workers.dev/policy/block-seo-tools.json)

```text
# AI Crawler Index — policy: block-seo-tools
# Block SEO and backlink crawlers
# Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic.
# Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/block-seo-tools.html
# 17 crawlers named. Paste into robots.txt at your document root.

User-agent: AhrefsBot
Disallow: /

User-agent: AhrefsSiteAudit
Disallow: /

User-agent: barkrowler
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-ESI
Disallow: /

User-agent: SemrushBot-FT
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: serpstatbot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: SplitSignalBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.pathwren.workers.dev/sitemap.xml
```

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/policy/block-seo-tools.html)
- [JSON](https://www.pathwren.workers.dev/policy/block-seo-tools.json)
- [Markdown](https://www.pathwren.workers.dev/policy/block-seo-tools.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/policy/block-seo-tools.html](https://www.pathwren.workers.dev/policy/block-seo-tools.html), generated from that page's own bytes in the same build. The HTML page is canonical.