Who actually crawls a small AI-agent site

163 named clients, 13448 requests, one host, 2026-08-31 20:58:11Z to 2026-09-03 06:25:38Z UTC (57.5 hours). One page each: the exact user-agent, when it came, how many addresses it came from, the paths it asked for in order, the status codes it got, and what it demanded that did not exist. Nothing here is inferred and nothing here is about intent.

curl -s https://www.pathwren.workers.dev/data/observed-clients.json   # all of it, one request
curl -s https://www.pathwren.workers.dev/bot/mcpbeat.json             # one client
curl -s https://www.pathwren.workers.dev/data/observed-clients.csv    # the table, flat

Why this exists

Most of these names have no page anywhere on the web. Search for Orbit-MCP-Registry-IconResolver or exaforce-mcprep and you will find a line in somebody's log and nothing that says what it fetched, in what order, or what it wanted that you do not serve. A host that publishes agent-facing documents gets visited by a class of machine that inspects other machines, and that traffic is invisible to every crawler list, because those lists document search and AI-training crawlers. This is the other kind, measured first-hand on one host.

Every client, by requests

What is not counted, and why

Our own traffic is not a client and is never counted. Of 40302 rows in the log: 26835 are ours (they send X-Self: 1, or name this loop in their user-agent) and 19 more were flagged by the analyst as one of our own agent runs reaching for one of our URLs through a model provider's fetch tool — those carry no header of ours and arrive from an address we cannot recognise, so they are excluded here by name rather than quietly kept. That leaves 13448 external requests, which is what every page here counts.

Also excluded from the dossiers, because there is no operator to name: 84 browser-shaped user-agents and 359 requests with no user-agent at all (9 address(es), 2026-08-31 23:41:45Z to 2026-09-03 06:24:14Z). They are listed here in full instead:

How it is measured

• One row per request, written server-side before anything is served — so clients that run no JavaScript are counted exactly like browsers. No beacon is required to appear here.
• Addresses are stored as salted hashes. This site publishes how many distinct ones a client used and never an address, in any form.
• A client is grouped by the documentation host it puts in its own user-agent; where it names none, by the product token it calls itself. Two names that cite nothing stay separate however similar they look, because merging them would be a guess.
• The class column is our classifier reading the user-agent. It is stated as ours because it has been wrong: it once booked ClaudeBot as an agent and a trust bot as a crawler.
• Every count carries its window. A count without one stops being true tomorrow.

Callable without an MCP client

The read-only tools this host runs over MCP answer a plain GET as well: no account, no key, no handshake, no JSON-RPC. Same implementation behind both doors — the HTTP endpoint calls the MCP tool in-process — so the two cannot drift.

curl -s 'https://www.pathwren.workers.dev/tools/whoami?&s=client-dossiers'

• /tools/whoami — Who is calling? — the one endpoint that needs nothing from you
• /tools/example — A worked example, run for real — no arguments
• /tools/verify-crawler — Does this address really belong to the crawler it claims to be?
• /tools/robots-lint — Lint a robots.txt
• /tools/robots-allowed — Would this crawler fetch this path?
• /tools/ai-access — Which AI crawlers does this file actually stop?
• /tools/classify-ua — What is this user-agent?
• /tools/verification-methods — Can this crawler be verified at all, and how?

All 8 of them, with a worked example each · as JSON.

The whole set as data

• /data/observed-clients.json — every dossier, one request.
• /data/observed-clients.csv — one row per client.
• /bot/index.json — the index above, machine-readable.
• /bot/index.md — the same page as markdown.
• /data/ua-regex.json — the observed_at_edge key is generated from this same corpus, so the regex and these pages cannot drift apart.
• /mcp/doctor — the document catalogue that names which of these clients asked for each discovery document, from the same log.

CC0. Rebuilt from the log on a timer, so tomorrow's clients appear by themselves and yesterday's counts stay true.