{"name":"ai-crawler-index","title":"AI Crawler Index — MCP server","version":"1.0.0","transport":"streamable-http","transport_docs":"https://www.pathwren.workers.dev/mcp-transport.html","endpoint":"https://www.pathwren.workers.dev/mcp","protocol_versions":["2026-07-28","2025-11-25","2025-06-18","2025-03-26","2024-11-05"],"stateless":true,"auth":"none — public, read-only, no key, no rate limit","call_this_first":{"tool":"whoami","takes_arguments":false,"invitation":"Takes no arguments. Safe to call. Deterministic. Touches no third party.","curl":"curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}'","without_json_rpc":"https://www.pathwren.workers.dev/tools/whoami?s=client-dossiers","also":"example — this server's own worked example, run end to end, no arguments either.","every_other_tool":"Every other tool here also accepts {}: an empty call returns that tool's documented worked example, labelled as one, instead of an error. Nothing here fetches a URL for you."},"tools":[{"name":"classify_user_agent","title":"Classify a user-agent string","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers"},{"name":"lookup_crawler","title":"Look up one crawler","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'."},{"name":"list_crawlers","title":"List and filter crawlers","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20."},{"name":"generate_robots_txt","title":"Generate a robots.txt for a stance","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'."},{"name":"is_verified_crawler_ip","title":"Check an IP against operator-published crawler ranges","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers"},{"name":"whats_changed","title":"What changed in the index","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'."},{"name":"changes_since","title":"What changed since your cursor","description":"FIRST CALL, needs nothing: {\"name\":\"whoami\",\"arguments\":{}} — Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'."},{"name":"whoami","title":"Who is calling? (no arguments)","description":"Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what the crawler index makes of the user-agent you sent, and whether the address you came from is inside a prefix its operator publishes — both run through this server's own classify_user_agent and is_verified_crawler_ip, on the same data a stranger can curl. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before."},{"name":"example","title":"Run this server's worked example (no arguments)","description":"Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — classify_user_agent on the exact user-agent string of the busiest client in this host's own published log (/data/observed-clients.json, 1164 requests in the window) — our data, and no third party is asked anything — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs classify_user_agent with {\"user_agent\":\"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1. and returns its real answer."}],"resources":[{"uri":"https://www.pathwren.workers.dev/data/agents.json","title":"The whole crawler dataset"},{"uri":"https://www.pathwren.workers.dev/data/ua-regex.json","title":"Compiled user-agent regexes"},{"uri":"https://www.pathwren.workers.dev/ip-ranges/all.json","title":"Every operator-published crawler prefix"},{"uri":"https://www.pathwren.workers.dev/status.json","title":"Mirror status"},{"uri":"https://www.pathwren.workers.dev/changes.json?since=0","title":"What changed since your cursor"}],"instructions":"THE FIRST CALL, COMPLETE. POST this body to https://www.pathwren.workers.dev/mcp — no key, no signup, no session id, nothing to fill in:\n\n  {\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}\n\nIt answers with what this host can see about you. Every one of the tools below also accepts arguments={} and returns its documented worked example rather than an error, so any of them can be the first one you try.\n\nSTART HERE IF YOU HAVE NOTHING TO HAND: `whoami` and `example` take no arguments at all — call either with {} and you get a real answer, not an error. whoami classifies the request you just sent (your user-agent, your address, whether this host has seen you before, what our robots policy says about you); example runs this server's own worked example end to end and returns exactly what a real call returns. Every other tool here also accepts {}: an empty call falls back to that tool's documented worked example, taken from this host's own data, and the answer says so. Nothing here fetches a URL for you.\n\nAI Crawler Index — a read-only index of the crawlers and AI agents that fetch the web, and of what blocking each one actually costs.\n\nTypical uses: turn a User-Agent header from a log into an identified crawler (classify_user_agent); decide a robots.txt policy and get the file (generate_robots_txt); verify that a request claiming to be GPTBot or Googlebot came from an address its operator publishes (is_verified_crawler_ip); watch for upstream changes (whats_changed), or read only what moved since your last call and nothing else (changes_since).\n\nTwo things to carry into any answer you build from these tools: a user-agent string is a claim and not a proof, and an address appearing in an operator's published range says the operator owns it, not that the traffic is welcome. Everything here is also plain HTTP at https://www.pathwren.workers.dev — no key, no rate limit, CC0.\n\nWHEN TO COME BACK. The 15 operator endpoints behind this data are re-read every 6 hours and nothing here can change faster than that: the next read is due at 2026-09-03T08:42:02+00:00. Call `changes_since` with the `cursor` from your last result and you get only what moved — it advances only on a real change, so an empty answer is proof rather than luck. Over plain HTTP the same delta is https://www.pathwren.workers.dev/changes.json?since=<cursor>, every machine document carries a strong ETag and a Last-Modified (If-None-Match and If-Modified-Since both answer 304 with no body), and `Cache-Control: max-age` counts down to that same instant. Polling faster is never rate limited and never refused; it just returns the cursor you already hold.\n\nTHE SAME TOOLS ON THE OTHER PROTOCOL, IF YOU SPEAK IT. This server is also an A2A (Agent2Agent) v1.0 agent at https://www.pathwren.workers.dev/a2a — every tool here is a skill there under the same name, answered by the same function, so the two cannot drift and the deploy is refused if they do. Its Agent Card is https://www.pathwren.workers.dev/.well-known/agent-card.json (the legacy agent.json spelling is served too). To invoke it, POST this complete body — no key, no session, no card to read first:\n\n  {\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}}\n\nIt answers with a Task already in state `completed` in the same response, so there is nothing to poll. Every skill on all eight A2A agents of this host as a ready-to-send body: https://www.pathwren.workers.dev/a2a/example.json — and both lanes with both directions in https://www.pathwren.workers.dev/documents.json under `rpc_endpoints`. If you only speak MCP, ignore all of this: everything is already here.","try_it":"curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"classify_user_agent\",\"arguments\":{\"user_agent\":\"GPTBot/1.2\"}}}'","data_behind_it":"https://www.pathwren.workers.dev/data/agents.json","license":"CC0-1.0","independent":true}