curl -s https://www.pathwren.workers.dev/crawler/semrushbot.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Semrush · SEO and backlink crawlers · json
User-agent: SemrushBot Disallow: /
| robots.txt token | SemrushBot |
| User-agent contains | SemrushBot |
| Operator | Semrush |
| Category | SEO and backlink crawlers |
| robots.txt | obeys robots.txt (documented) |
| Verify by | reverse DNS |
Semrush's backlink and keyword crawler. It is not an AI crawler, but it is usually in the top three by volume on any site, and it is the cheapest block on this list.
Your competitors' Semrush reports get thinner, and so do yours. No user-facing effect.
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
User-agent: SemrushBot Allow: /
Operator documentation: https://www.semrush.com/bot/
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-seo-tools