archive.org_bot

What this client asked www.pathwren.workers.dev for, when, and what it got. Every number below is from this host's own server-side request log, 2026-08-31 20:58:34Z to 2026-09-03 03:05:14Z UTC.

Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/d3712ae warc/v0.8.99

993 request(s) from 7 distinct address(es), 898 distinct path(s), first seen 2026-08-31 20:58:34Z, last seen 2026-09-03 03:05:14Z UTC. 8 separate visit(s), counting a gap of more than 30 minutes as a new one, median 360.6 minutes between them.

It describes itself, inside its own user-agent, as archive.org_bot +http://archive.org/details/archive.org_bot. That is the client's own words, quoted; this page makes no claim about what it is for.

The crawler catalogue on this site has a record for it: /crawler/archive-org-bot.html — what it is for, and what blocking it costs. This page is only what it did here.

The user-agent strings, exactly as they arrived

user-agentrequestsaddressesfirst seenlast seen
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/d3712ae warc/v0.8.9999372026-08-31 20:58:34Z2026-09-03 03:05:14Z

What it asked for, in the order it asked

First request to each path, oldest first, first 40 shown.

#pathat (UTC)status
1/category/dataset.html2026-08-31 20:58:34Znot recorded
2/category/archive.html2026-08-31 20:58:34Znot recorded
3/robots/block-disputed.txt2026-08-31 20:58:34Znot recorded
4/crawler/oai-searchbot.html2026-08-31 20:58:34Znot recorded
5/status.html2026-08-31 20:58:34Znot recorded
6/robots/allow-ai-search-only.txt2026-08-31 20:58:34Znot recorded
7/category/user-fetch.html2026-08-31 20:58:34Znot recorded
8/data/2026-08-31 20:58:34Znot recorded
9/robots/maximum-ai-visibility.txt2026-08-31 20:58:34Znot recorded
10/crawler/ccbot.html2026-08-31 20:58:34Znot recorded
11/category/seo.html2026-08-31 20:58:34Znot recorded
12/policy/block-disputed.html2026-08-31 20:58:34Znot recorded
13/category/tool.html2026-08-31 20:58:34Znot recorded
14/crawler/gptbot.html2026-08-31 20:58:34Znot recorded
15/crawler/perplexitybot.html2026-08-31 20:58:34Znot recorded
16/snippet/2026-08-31 20:58:34Znot recorded
17/ip-ranges/2026-08-31 20:58:34Znot recorded
18/policy/allow-ai-search-only.html2026-08-31 20:58:34Znot recorded
19/crawler/claude-searchbot.html2026-08-31 20:58:34Znot recorded
20/policy/block-all-ai.html2026-08-31 20:58:34Znot recorded
21/ip-ranges/index.html2026-08-31 20:58:34Znot recorded
22/px.gif2026-08-31 20:58:35Znot recorded
23/data/agents.json2026-08-31 20:58:35Z200
24/data/index.html2026-08-31 20:58:39Znot recorded
25/ip-ranges/openai-chatgpt-user.json2026-08-31 20:58:41Znot recorded
26/crawler/chatgpt-user.html2026-08-31 20:58:41Znot recorded
27/crawler/gptbot.json2026-08-31 20:58:41Znot recorded
28/crawler/oai-searchbot.json2026-08-31 20:58:41Znot recorded
29/ip-ranges/openai-gptbot.json2026-08-31 20:58:41Znot recorded
30/ip-ranges/openai-searchbot.json2026-08-31 20:58:41Znot recorded
31/crawler/chatgpt-user.json2026-08-31 20:58:41Znot recorded
32/crawler/claudebot.html2026-08-31 20:58:41Znot recorded
33/data/Mozilla/5.0%20(compatible;%20ClaudeBot/1.0;%20+claudebot@anthropic.com)2026-08-31 20:58:43Znot recorded
34/crawler/claudebot.json2026-08-31 20:58:43Znot recorded
35/crawler/claude-searchbot.json2026-08-31 20:58:43Znot recorded
36/data/Mozilla/5.0%20(compatible;%20Claude-SearchBot/1.0;%20+Claude-SearchBot@anthropic.com)2026-08-31 20:58:43Znot recorded
37/data/Mozilla/5.0%20(compatible;%20Claude-User/1.0;%20+Claude-User@anthropic.com)2026-08-31 20:58:43Znot recorded
38/crawler/claude-user.html2026-08-31 20:58:43Znot recorded
39/crawler/claude-user.json2026-08-31 20:58:43Znot recorded
40/crawler/anthropic-ai.html2026-08-31 20:58:43Znot recorded

Everything it asked for

pathrequestsstatus codesfirstlast
/px.gif63not recorded×45, 200×182026-08-31 20:58:35Z2026-09-03 03:05:14Z
/changes.json4200×42026-09-02 03:03:21Z2026-09-02 03:03:26Z
/about.html3200×2, not recorded×12026-08-31 20:59:44Z2026-09-02 09:04:26Z
/bot/agentgrade.html3200×32026-09-01 07:02:31Z2026-09-01 09:00:15Z
/bot/gptbot.html3200×32026-09-01 07:02:34Z2026-09-01 15:00:55Z
/crawler/bytespider.html3200×2, not recorded×12026-08-31 20:59:01Z2026-09-02 09:03:16Z
/crawler/google-extended.html3200×2, not recorded×12026-08-31 20:58:46Z2026-09-02 09:03:16Z
/data/agents.json3200×2, not recorded×12026-08-31 20:58:35Z2026-09-02 09:03:19Z
/openapi.json3200×2, not recorded×12026-08-31 20:59:36Z2026-09-02 09:03:16Z
/bot/node.html2200×22026-09-01 07:02:16Z2026-09-01 07:02:40Z
/bot/yandexbot.html2200×22026-09-01 07:02:17Z2026-09-01 07:02:20Z
/crawler/ccbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:59:01Z
/crawler/claude-searchbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:43Z
/crawler/gptbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:41Z
/crawler/oai-searchbot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:41Z
/crawler/perplexitybot.html2not recorded×22026-08-31 20:58:34Z2026-08-31 20:58:59Z
/mcp.html2200×22026-09-01 03:00:04Z2026-09-01 09:00:15Z
/tools/ai-access.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:57Z
/tools/classify-ua.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/example.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:57Z
/tools/robots-allowed.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/robots-lint.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/verification-methods.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/verify-crawler.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:58Z
/tools/whoami.html2200×22026-09-02 09:03:20Z2026-09-02 09:04:57Z
/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/.well-known/ai-catalog.json1200×12026-09-01 09:00:09Z2026-09-01 09:00:09Z
/.well-known/ard.json1200×12026-09-01 09:00:09Z2026-09-01 09:00:09Z
/.well-known/owners.json1200×12026-09-01 09:00:14Z2026-09-01 09:00:14Z
/a2a1200×12026-09-02 09:03:14Z2026-09-02 09:03:14Z
/a2a.html1200×12026-09-01 09:00:15Z2026-09-01 09:00:15Z
/a2a/doctor/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/example.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/netcheck/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/robots/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/a2a/triage/.well-known/agent-card.json1200×12026-09-02 09:03:20Z2026-09-02 09:03:20Z
/about.json1200×12026-09-02 09:04:58Z2026-09-02 09:04:58Z
/api.html1200×12026-09-01 07:02:16Z2026-09-01 07:02:16Z
/apple-touch-icon.png1200×12026-09-01 03:00:05Z2026-09-01 03:00:05Z
/bot/402explorer.html1200×12026-09-01 07:02:40Z2026-09-01 07:02:40Z
/bot/AgentGrade/1.0%20(security%20research%3B%20agentgrade.net)1404×12026-09-01 09:00:15Z2026-09-01 09:00:15Z
/bot/AgentGrade/1.0%20(security%20research;%20agentgrade.net)1307×12026-09-01 09:00:14Z2026-09-01 09:00:14Z
/bot/GPTBot/1.41404×12026-09-01 15:00:55Z2026-09-01 15:00:55Z
/bot/YandexBot/3.01404×12026-09-01 07:02:22Z2026-09-01 07:02:22Z
/bot/a2a-directory-discovery.html1200×12026-09-01 07:02:31Z2026-09-01 07:02:31Z
/bot/a2a-directory-liveness.html1200×12026-09-01 07:02:35Z2026-09-01 07:02:35Z
/bot/a2a-registry-healthcheck.html1200×12026-09-01 07:02:33Z2026-09-01 07:02:33Z
/bot/a2a-registry-smoke.html1200×12026-09-01 07:02:19Z2026-09-01 07:02:19Z
/bot/a2a-registry.html1200×12026-09-01 07:02:27Z2026-09-01 07:02:27Z
/bot/aetherlink-public-agent-card-policy-check.html1200×12026-09-01 07:02:26Z2026-09-01 07:02:26Z
/bot/aetherlink-public-discovery-evidence.html1200×12026-09-01 07:02:31Z2026-09-01 07:02:31Z
/bot/aetherlinkdiscoveryevidence.html1200×12026-09-01 07:02:42Z2026-09-01 07:02:42Z
/bot/agenstrybot.html1200×12026-09-01 07:02:37Z2026-09-01 07:02:37Z
/bot/agent-guild-scout.html1200×12026-09-01 07:02:41Z2026-09-01 07:02:41Z
/bot/agent-tools-cloud-crawler.html1200×12026-09-01 07:02:34Z2026-09-01 07:02:34Z
/bot/agentdisco.html1200×12026-09-01 07:02:30Z2026-09-01 07:02:30Z
/bot/agentgrade.json1200×12026-09-01 09:00:09Z2026-09-01 09:00:09Z
/bot/agentgrade.md1200×12026-09-01 09:00:09Z2026-09-01 09:00:09Z
/bot/agentindexbot.html1200×12026-09-01 07:02:26Z2026-09-01 07:02:26Z
/bot/agentreputationbot.html1200×12026-09-01 07:02:24Z2026-09-01 07:02:24Z

838 further paths are in the JSON.

What it asked for that did not exist

path it asked forwhat it gotfirst askedsince then
/data/A2A-Registry-HealthCheck/1.0404×12026-09-01 07:02:17Zstill absent
/data/bingbot/2.0404×12026-09-01 07:02:17Zstill absent
/data/ClaudeBot/1.0404×12026-09-01 07:02:17Zstill absent
/data/exaforce-mcprep/0.1%20(MCP%20server%20reputation%20scanner)404×12026-09-01 07:02:17Zstill absent
/data/python-httpx2/2.12.0404×12026-09-01 07:02:18Zstill absent
/data/YandexBot/3.0404×12026-09-01 07:02:18Zstill absent
/data/Orbit-MCP-Registry-IconResolver/1.0404×12026-09-01 07:02:18Zstill absent
/data/Mozilla/5.0404×12026-09-01 07:02:18Zstill absent
/data/Go-http-client/2.0404×12026-09-01 07:02:18Zstill absent
/data/mcp-registry/1.0404×12026-09-01 07:02:18Zstill absent
/data/GPTBot/1.4404×12026-09-01 07:02:18Zstill absent
/data/measure-mcp-schema/0.1.0404×12026-09-01 07:02:18Zstill absent
/data/mcp-checker/1.0404×12026-09-01 07:02:18Zstill absent
/data/GF-Agent-Toll-Outbound/1.6.0404×12026-09-01 07:02:18Zstill absent
/data/utopian-foundry-probe/1.0404×12026-09-01 07:02:18Zstill absent
/data/MCPCatalogSync/1.0404×12026-09-01 07:02:18Zstill absent
/data/GF-Agent-Toll-Remediation/1.0404×12026-09-01 07:02:18Zstill absent
/data/AetherLink-Public-Discovery-Evidence/1.0404×12026-09-01 07:02:18Zstill absent
/data/python-httpx/0.28.1404×12026-09-01 07:02:18Zstill absent
/data/ReactorNetty/1.0.32404×12026-09-01 07:02:18Zstill absent
/data/ReactorNetty/1.2.1404×12026-09-01 07:02:19Zstill absent
/data/A2A-Registry/1.0404×12026-09-01 07:02:19Zstill absent
/data/AetherLink-Public-Agent-Card-Policy-Check/1.0404×12026-09-01 07:02:19Zstill absent
/data/AgentSure-MCPScan/0.1404×12026-09-01 07:02:19Zstill absent
/data/curl/7.88.0404×12026-09-01 07:02:19Zstill absent
/data/PackageHound/1.0404×12026-09-01 07:02:19Zstill absent
/data/SaSame-Census-Era-Probe/1.0404×12026-09-01 07:02:19Zstill absent
/data/SaSame-MCP-Audit/0.1404×12026-09-01 07:02:19Zstill absent
/data/TAR-Discovery/1.0404×12026-09-01 07:02:19Zstill absent
/data/TAR-Directory-Indexer/1.0404×12026-09-01 07:02:19Zstill absent
/data/TAR-Health/1.0404×12026-09-01 07:02:19Zstill absent
/bot/YandexBot/3.0404×12026-09-01 07:02:22Zstill absent
/bot/text/html%2Capplication/xhtml%2Bxml%2Capplication/xml%3Bq%3D0.9%2C*/*%3Bq%3D0.8404×12026-09-01 07:02:22Zstill absent
/data/APIEvangelist-AddAPI-Control/1.0%20(%2Bhttps%3A/apis.io)404×12026-09-01 07:02:48Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20archive.org_bot%20%2Bhttp%3A/archive.org/details/archive.org_bot)%20Zeno/d3712ae%20warc/v0.8.99404×12026-09-01 07:02:48Zstill absent
/data/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20ClaudeBot/1.0%3B%20%2Bclaudebot%40anthropic.com)404×12026-09-01 07:02:48Zstill absent
/data/text/html%2Capplication/xhtml%2Bxml%2Capplication/xml%3Bq%3D0.9%2C*/*%3Bq%3D0.8404×12026-09-01 07:02:48Zstill absent
/data/apis.io-submit/1.0%20(%2Bhttps%3A/apis.io)404×12026-09-01 07:02:48Zstill absent
/data/Mozilla/5.0%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20bingbot/2.0%3B%20%2Bhttp%3A/www.bing.com/bingbot.htm)%20Chrome/116.0.1938.76%20Safari/537.36404×12026-09-01 07:02:48Zstill absent
/data/AgentGrade/1.0%20(security%20research%3B%20agentgrade.net)404×12026-09-01 07:02:48Zstill absent
/data/agent-tools.cloud-crawler/0.1%20(%2Bhttps%3A/agent-tools.cloud)404×12026-09-01 07:02:48Zstill absent
/data/Deno/2.1.4%20(variant%3B%20SupabaseEdgeRuntime/1.74.3)404×12026-09-01 07:02:48Zstill absent
/data/lastseen-schema-probe/1.0%20(%2Bhttps%3A/lastseen.dev%3B%20introspection-only)404×12026-09-01 07:02:48Zstill absent
/data/hultra-link/1.0%20(%2Bhttps%3A/donnees.hultra.link/sondes.md)404×12026-09-01 07:02:48Zstill absent
/data/agent-tools.cloud-a2a/0.1%20(%2Bhttps%3A/agent-tools.cloud)404×12026-09-01 07:02:48Zstill absent
/data/application/json%2C%20text/event-stream%3Bq%3D0.9%2C%20*/*%3Bq%3D0.1404×12026-09-01 07:02:48Zstill absent
/data/MCPWatch/0.1.0%20(%2Bmcpwatch%40iyre.com)%20longitudinal%20MCP%20security%20research404×12026-09-01 07:02:48Zstill absent
/data/AgentReputationBot/1.0%20(%2Bhttps%3A/agentreputation.dev)404×12026-09-01 07:02:49Zstill absent
/data/mcp-schema-archive/1.0%20(%2Bhttps%3A/mcp-schema-archive.delfrost42.workers.dev)404×12026-09-01 07:02:49Zstill absent
/data/AgentIndexBot/0.1%20(%2Bhttps%3A/agents.traderszone.net%3B%20polite%20ARD%20crawler)404×12026-09-01 07:02:49Zstill absent
/data/x402-observatory/0.2%20(%2Bresearch%20collector%3B%20contact%3A%20300tsb%40gmail.com)404×12026-09-01 07:02:49Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20MCPCatalogSync/1.0)404×12026-09-01 07:02:49Zstill absent
/data/agent-guild-scout/1%20(%2Bhttps%3A/agent-guild-5d5r.onrender.com/.well-known/agent-guild.json)404×12026-09-01 07:02:49Zstill absent
/data/A2A-Registry-Smoke/1.0%20(%2Bhttps%3A/a2aregistry.org)404×12026-09-01 07:02:49Zstill absent
/data/mcp-observatory/0.1.0%20(%2Bhttps%3A/github.com/yhouta/mcp-observatory%3B%20public%20transparency%20log%3B%20contact%3A%20yassine.houta%40outlook.fr)404×12026-09-01 07:02:49Zstill absent
/data/SaSameAgentAudit/0.1%20(%2Bhttps%3A/live-vps.sasame.online/.well-known/agent-card.json)404×12026-09-01 07:02:49Zstill absent
/data/pip/24.3.1%20%7B%22ci%22%3Anull%2C%22cpu%22%3A%22x86_64%22%2C%22implementation%22%3A%7B%22name%22%3A%22CPython%22%2C%22version%22%3A%223.11.9%22%7D%2C%22installer%22%3A%7B%22name%22%3A%22pip%22%2C%22version%22%3A%2224.3.1%22%7D%2C%22openssl_version%22%3A%22OpenSSL%203.0.13%2030%20Jan%202024%22%2C%22python%22%3A%223.11.9%22%2C%22setuptools_version%22%3A%2275.6.0%22%2C%22system%22%3A%7B%22name%22%3A%22Linux%22%2C%22release%22%3A%226.8.0%22%7D%7D404×12026-09-01 07:02:49Zstill absent
/data/truespar-mcp-registry/0.1%20(%2Bhttps%3A/registry.truespar.com)404×12026-09-01 07:02:49Zstill absent
/data/application/json%2C%20application/a2a%2Bjson%3Bq%3D0.9404×12026-09-01 07:02:49Zstill absent
/data/AetherLinkDiscoveryEvidence/1.0%20(%2Bhttps%3A/aetherlink.47-85-186-77.sslip.io/llms.txt)404×12026-09-01 07:02:50Zstill absent
/data/api-forge-mcp-index/1.0%20(%2Bhttps%3A/api.temsor.com/mcp/index%3B%20iletisim%3A%20altyapi%40temsor.com)404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20AgentSure-MCPScan/0.1%3B%20%2Bhttps%3A/agentsure.tech)404×12026-09-01 07:02:50Zstill absent
/data/application/json%2Ctext/html%3Bq%3D0.9%2C*/*%3Bq%3D0.7404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/151.0.0.0%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(iPhone%3B%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML%2C%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(X11%3B%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/148.0.0.0%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/124.0.0.0%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Linux%3B%20Android%2010%3B%20K)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/151.0.0.0%20Mobile%20Safari/537.36404×12026-09-01 07:02:50Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/125.0.0.0%20Safari/537.36%20Edg/125.0.0.0404×12026-09-01 07:02:51Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/144.0.0.0%20Safari/537.36404×12026-09-01 07:02:51Zstill absent
/data/Mozilla/5.0%20(Windows%20NT%2010.0)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko)%20Chrome/106.0.0.0%20Safari/537.36404×12026-09-01 07:02:51Zstill absent
/data/Mozilla/5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_1)%20AppleWebKit/600.2.5%20(KHTML%2C%20like%20Gecko)%20Version/8.0.2%20Safari/600.2.5%20(Gort)404×12026-09-01 07:02:51Zstill absent
/bot/AgentGrade/1.0%20(security%20research%3B%20agentgrade.net)404×12026-09-01 09:00:15Zstill absent
/bot/GPTBot/1.4404×12026-09-01 15:00:55Zstill absent
/data/Googlebot-Video/1.0404×12026-09-02 09:03:16Zstill absent
/data/GoogleOther-Image/1.0404×12026-09-02 09:03:18Zstill absent
/data/GoogleOther-Video/1.0404×12026-09-02 09:03:20Zstill absent
/crawler/Googlebot-Video/1.0404×12026-09-02 09:04:26Zstill absent
/crawler/GoogleOther-Image/1.0404×12026-09-02 09:04:26Zstill absent
/crawler/GoogleOther-Video/1.0404×12026-09-02 09:04:27Zstill absent
/data/Mozilla/5.0%20(Linux%3B%20Android%205.0%3B%20SM-G920A)%20AppleWebKit%20(KHTML%2C%20like%20Gecko)%20Chrome%20Mobile%20Safari%20(compatible%3B%20AdsBot-Google-Mobile%3B%20%2Bhttp%3A/www.google.com/mobile/adsbot.html)404×12026-09-02 09:04:29Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20Google-Site-Verification/1.0)404×12026-09-02 09:04:31Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20Google-CWS)404×12026-09-02 09:04:32Zstill absent
/data/Mozilla/5.0%20(X11%3B%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20Google-Agent%3B%20%2Bhttps%3A/developers.google.com/crawling/docs/crawlers-fetchers/google-agent)%20Chrome/W.X.Y.Z%20Safari/537.36404×12026-09-02 09:04:33Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SiteAuditBot/0.97%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-BA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-SI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-SWA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:37Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SplitSignalBot%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:38Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-FT%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:38Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SemrushBot-ESI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:38Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20DotBot/1.2%3B%20%2Bhttps%3A/opensiteexplorer.org/dotbot%3B%20help%40moz.com)404×12026-09-02 09:04:40Zstill absent
/data/rogerbot/1.2%20(http%3A/moz.com/help/pro/what-is-rogerbot-%2C%20rogerbot-crawler%40moz.com)404×12026-09-02 09:04:40Zstill absent
/data/serpstatbot/2.1%20(advanced%20backlink%20tracking%20bot%3B%20https%3A/serpstatbot.com/%3B%20abuse%40serpstatbot.com)404×12026-09-02 09:04:41Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20SEOkicks%3B%20%2Bhttps%3A/www.seokicks.de/robot.html)404×12026-09-02 09:04:41Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20MojeekBot/0.11%3B%20%2Bhttps%3A/www.mojeek.com/bot.html)404×12026-09-02 09:04:41Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20Pinterestbot/1.0%3B%20%2Bhttps%3A/www.pinterest.com/bot.html)404×12026-09-02 09:04:43Zstill absent
/data/Mozilla/5.0%20(compatible%3B%20panscient.com)404×12026-09-02 09:04:44Zstill absent
/data/AwarioSmartBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:50Zstill absent
/data/AwarioRssBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:51Zstill absent
/crawler/Mozilla/5.0%20(Linux%3B%20Android%205.0%3B%20SM-G920A)%20AppleWebKit%20(KHTML%2C%20like%20Gecko)%20Chrome%20Mobile%20Safari%20(compatible%3B%20AdsBot-Google-Mobile%3B%20%2Bhttp%3A/www.google.com/mobile/adsbot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20Google-Site-Verification/1.0)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(X11%3B%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20Google-Agent%3B%20%2Bhttps%3A/developers.google.com/crawling/docs/crawlers-fetchers/google-agent)%20Chrome/W.X.Y.Z%20Safari/537.36404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20Google-CWS)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SiteAuditBot/0.97%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-SWA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-BA%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-SI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SplitSignalBot%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-ESI%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20DotBot/1.2%3B%20%2Bhttps%3A/opensiteexplorer.org/dotbot%3B%20help%40moz.com)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SemrushBot-FT%3B%20%2Bhttp%3A/www.semrush.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/rogerbot/1.2%20(http%3A/moz.com/help/pro/what-is-rogerbot-%2C%20rogerbot-crawler%40moz.com)404×12026-09-02 09:04:58Zstill absent
/crawler/serpstatbot/2.1%20(advanced%20backlink%20tracking%20bot%3B%20https%3A/serpstatbot.com/%3B%20abuse%40serpstatbot.com)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20SEOkicks%3B%20%2Bhttps%3A/www.seokicks.de/robot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20MojeekBot/0.11%3B%20%2Bhttps%3A/www.mojeek.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20Pinterestbot/1.0%3B%20%2Bhttps%3A/www.pinterest.com/bot.html)404×12026-09-02 09:04:58Zstill absent
/crawler/AwarioRssBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:58Zstill absent
/crawler/AwarioSmartBot/1.0%20(%2Bhttps%3A/awario.com/bots.html%3B%20bots%40awario.com)404×12026-09-02 09:04:58Zstill absent
/crawler/Mozilla/5.0%20(compatible%3B%20panscient.com)404×12026-09-02 09:04:58Zstill absent

What it got, and what it sent

Status codes
200×483, not recorded×309, 404×120, 307×81
Accept headers
none sent
Bytes served
4579706
Attributed to a channel
direct (45), mcp-registry-official (4), ard (3)
Our instrument classed it
crawler×993 — that is our classifier's label from the user-agent, not the operator's, and it has been wrong before.

How to verify it is really them

no published verification method

From the crawler record at /crawler/archive-org-bot.html, which cites https://archive.org/details/archive.org_bot — one source for both pages.

What it publishes about you afterwards

Not observed. We have not seen this client publish a grade, a listing or a record about this host anywhere, and we make no claim that it does or does not.

Its own documentation

It names http://archive.org/details/archive.org_bot in its own user-agent. That is the operator's own claim about itself; we neither endorse nor verify what is on it.

This page as data

curl -s https://www.pathwren.workers.dev/bot/archive-org-bot.json
curl -s https://www.pathwren.workers.dev/data/observed-clients.json   # every client, one request

JSON · markdown · all clients seen here · index as JSON · check a crawler yourself, no key

Method, and what this page will not say

Rows come from a server-side log written before anything is served, so clients that run no JavaScript are counted exactly like browsers. The window is the whole life of the log, 2026-08-31 20:58:34Z to 2026-09-03 03:05:14Z UTC, and it is stated on every number because a count without a window is not a fact. Addresses are stored as salted hashes and counted, never printed. Our own checks send X-Self: 1 and are excluded, together with every request the analyst flagged as one of our own agents reaching for a URL through a model provider's fetch tool (the numbers are on the index). Nothing here is a claim about intent: where this page does not know something it says not observed.