# SemrushBot

What this client asked https://www.pathwren.workers.dev for, from that host's own request log.
Window: 2026-09-03 20:46:28Z to 2026-09-05 17:08:21Z UTC. Generated 2026-09-06.

## User-agent strings, exactly as they arrived

- `Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)` — 538 request(s), 39 address(es), 2026-09-03 20:46:28Z to 2026-09-05 17:08:21Z

- Requests: 538
- Distinct addresses (salted hashes, never published): 39
- Distinct paths: 509
- Visits (30-minute gap rule): 4
- Status codes: 200×538
- Accept headers: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml, */*
- Our classifier called it: crawler×538 (ours, from the user-agent)

## What it asked for, in order

1. `/robots.txt` — 2026-09-03 20:46:28Z — 200
2. `/` — 2026-09-03 20:46:29Z — 200
3. `/index.html` — 2026-09-03 20:46:32Z — 200
4. `/sitemap.xml` — 2026-09-03 20:46:34Z — 200
5. `/crawler/icc-crawler.html` — 2026-09-04 20:56:48Z — 200
6. `/crawler/duckduckbot.html` — 2026-09-04 21:09:07Z — 200
7. `/crawler/google-extended.html` — 2026-09-04 21:17:44Z — 200
8. `/crawler/mediapartners-google.html` — 2026-09-04 21:27:42Z — 200
9. `/operator/parallel.html` — 2026-09-04 21:40:15Z — 200
10. `/crawler/seznambot.html` — 2026-09-04 21:43:50Z — 200
11. `/bot/agoragentic-safefetch.html` — 2026-09-04 21:50:24Z — 200
12. `/operator/lyrenth.html` — 2026-09-04 21:52:05Z — 200
13. `/crawler/yandexadditional.html` — 2026-09-04 21:52:21Z — 200
14. `/bot/mcp2-research.html` — 2026-09-04 21:56:41Z — 200
15. `/operator/bytedance.html` — 2026-09-04 21:57:53Z — 200
16. `/bot/duckduckbot.html` — 2026-09-04 21:58:07Z — 200
17. `/bot/truespar-mcp-registry.html` — 2026-09-04 21:58:26Z — 200
18. `/pricing` — 2026-09-04 22:13:16Z — 200
19. `/bot/rootz-mcp-registry-prober.html` — 2026-09-04 22:14:50Z — 200
20. `/operator/mistral.html` — 2026-09-04 22:15:43Z — 200
21. `/bot/movanas-registry-snapshot.html` — 2026-09-04 22:19:23Z — 200
22. `/operator/yandex.html` — 2026-09-04 22:19:45Z — 200
23. `/crawler/pinterestbot.html` — 2026-09-04 22:22:11Z — 200
24. `/operator/timpi.html` — 2026-09-04 22:23:42Z — 200
25. `/bot/qtcreator.html` — 2026-09-04 22:25:08Z — 200
26. `/category/ai-training.html` — 2026-09-04 22:26:15Z — 200
27. `/crawler/semrushbot-ocob.html` — 2026-09-04 22:28:26Z — 200
28. `/bot/conway-replicatio-r91-strict-a2a-probe.html` — 2026-09-04 22:28:57Z — 200
29. `/bot/reactornetty.html` — 2026-09-04 22:30:51Z — 200
30. `/bot/discordbot.html` — 2026-09-04 22:32:01Z — 200
31. `/crawler/mistralai-user.html` — 2026-09-04 22:32:43Z — 200
32. `/ip-ranges/openai-searchbot.html` — 2026-09-04 22:35:12Z — 200
33. `/crawler/tiktokspider.html` — 2026-09-04 22:35:30Z — 200
34. `/bot/mcpi.html` — 2026-09-04 22:40:58Z — 200
35. `/bot/actablesitebot.html` — 2026-09-04 22:41:22Z — 200
36. `/crawler/bytespider.html` — 2026-09-04 22:41:28Z — 200
37. `/ai-crawler-logs/index.html` — 2026-09-04 22:44:59Z — 200
38. `/crawler/img2dataset.html` — 2026-09-04 22:48:22Z — 200
39. `/operator/duckduckgo.html` — 2026-09-04 22:48:46Z — 200
40. `/bot/ping-blo-gs.html` — 2026-09-04 22:49:10Z — 200

## What it asked for that did not exist

Nothing.

## Verification

reverse DNSFrom the crawler record at /crawler/semrushbot.html, which cites https://www.semrush.com/bot/ — one source for both pages.

## Ask this page a question, with one GET

A question about this page, answered by a call you can make right now — no key, no account, no JSON-RPC, nothing to assemble. The query string is already filled in with this page's own subject.

- **What is Mozilla/5.0 (compatible; SemrushBot/7~bl; +ht…?**  
  `curl -s 'https://www.pathwren.workers.dev/tools/classify-ua?ua=Mozilla%2F5.0%20%28compatible%3B%20SemrushBot%2F7~bl%3B%20%2Bhttp%3A%2F%2Fwww.semrush.com%2Fbot.html%29&s=client-dossiers'`  
  This exact user-agent run through the catalogue: which crawler it is, who operates it, what the crawl is for, what blocking it costs — or an honest "not recognised" if it is nobody we know.

Every parameter above is a value the endpoint reads, not an address it visits: nothing is fetched on your behalf. Called without any parameter, each endpoint returns its own specification instead of an error.


## What it publishes about you

Not observed. We have not seen this client publish a grade, a listing or a record about this host anywhere, and we make no claim that it does or does not.

## Documentation it names

http://www.semrush.com/bot.html

JSON: https://www.pathwren.workers.dev/bot/semrushbot.json · all clients: https://www.pathwren.workers.dev/data/observed-clients.json