curl -s https://www.pathwren.workers.dev/category/seo.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| AhrefsBot | AhrefsBot | Ahrefs | No user-facing effect. Ahrefs honours Crawl-delay, so rate-limiting is usually better than… |
| AhrefsSiteAudit | AhrefsSiteAudit | Ahrefs | Site owners auditing your domain get an incomplete report. Blocking it saves bandwidth and… |
| Barkrowler | barkrowler | Babbar | You leave Babbar's index. No effect on search or assistants.… |
| DataForSeoBot | DataForSeoBot | DataForSEO | You leave a dataset that a long tail of SEO products is built on. No user-facing effect.… |
| DotBot | dotbot | Moz | You leave Moz's link index, so Domain Authority and link reports about your site get thinn… |
| MJ12bot | MJ12bot | Majestic | You leave the Majestic backlink index. No search or AI effect. It supports Crawl-delay, wh… |
| rogerbot | rogerbot | Moz | Moz Pro site audits of your domain stop. If the domain is yours and you use Moz, blocking … |
| SemrushBot | SemrushBot | Semrush | Your competitors' Semrush reports get thinner, and so do yours. No user-facing effect.… |
| SemrushBot-BA | SemrushBot-BA | Semrush | None to you. It costs the site being audited a little accuracy in their backlink report.… |
| SemrushBot-ESI | SemrushBot-ESI | Semrush | Enterprise customers lose analysis of your domain. Nothing user-facing.… |
| SemrushBot-FT | SemrushBot-FT | Semrush | Your text stops being compared against other people's submissions — which also means copie… |
| SemrushBot-SI | SemrushBot-SI | Semrush | Nothing user-facing. Semrush customers lose on-page suggestions for pages on your domain.… |
| SemrushBot-SWA | SemrushBot-SWA | Semrush | Writers using Semrush's assistant see your links reported as unreachable.… |
| SEOkicks | SEOkicks | SEOkicks | You leave a regional backlink index. Nothing else changes.… |
| serpstatbot | serpstatbot | Serpstat | You leave Serpstat's link index. Try Crawl-delay first — they honour it, and a slow crawle… |
| SiteAuditBot | SiteAuditBot | Semrush | Site Audit reports on your own domain stop working. If a customer is auditing your site wi… |
| SplitSignalBot | SplitSignalBot | Semrush | A site owner's own A/B testing stops. Only relevant on domains whose owner uses the produc… |