curl -s https://www.pathwren.workers.dev/policy/block-seo-tools.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic.
curl -s https://www.pathwren.workers.dev/robots/block-seo-tools.txt >> robots.txt
The cheapest bandwidth saving available to most sites, and the one nobody regrets. The only cost is that your own dashboards on those tools get thinner. Consider Crawl-delay first: both major operators honour it.
AhrefsBot · AhrefsSiteAudit · Barkrowler · DataForSeoBot · DotBot · MJ12bot · rogerbot · SemrushBot · SemrushBot-BA · SemrushBot-ESI · SemrushBot-FT · SemrushBot-SI · SemrushBot-SWA · SEOkicks · serpstatbot · SiteAuditBot · SplitSignalBot
/robots/block-seo-tools.txt · json
# AI Crawler Index — policy: block-seo-tools # Block SEO and backlink crawlers # Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic. # Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/block-seo-tools.html # 17 crawlers named. Paste into robots.txt at your document root. User-agent: AhrefsBot Disallow: / User-agent: AhrefsSiteAudit Disallow: / User-agent: barkrowler Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: dotbot Disallow: / User-agent: MJ12bot Disallow: / User-agent: rogerbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-ESI Disallow: / User-agent: SemrushBot-FT Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: SEOkicks Disallow: / User-agent: serpstatbot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: SplitSignalBot Disallow: / User-agent: * Allow: / Sitemap: https://www.pathwren.workers.dev/sitemap.xml