{
 "name": "MCP server — AI Crawler Index",
 "what": "This index as an MCP server: Streamable HTTP at /mcp, nine read-only tools, no key, no signup. Classify a user-agent, look up a crawler, generate robots.txt, check an IP.",
 "url": "https://www.pathwren.workers.dev/mcp.json",
 "twin_of": "https://www.pathwren.workers.dev/mcp.html",
 "page": {
  "path": "/mcp.html",
  "url": "https://www.pathwren.workers.dev/mcp.html",
  "type": "text/html"
 },
 "generated_at": "2026-09-03T06:29:54+00:00",
 "generated_from": "the bytes of /mcp.html, by surfaces/ai-crawler-index/build.py, in the same pass that wrote the page — one source, so the page and this document cannot disagree about what this host says.",
 "license": {
  "document": "CC0-1.0",
  "url": "https://creativecommons.org/publicdomain/zero/1.0/"
 },
 "access": {
  "api_key": "none",
  "account": "none",
  "rate_limit": "none",
  "cors": "*",
  "auth": "none — every document here is a public GET"
 },
 "commands": [
  "# it answers right now — no account, no header ceremony",
  "curl -s https://www.pathwren.workers.dev/mcp \\",
  "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
  "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}' | jq -r '.result.tools[].name'",
  "claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp",
  "{",
  " \"mcpServers\": {",
  "  \"ai-crawler-index\": { \"type\": \"streamable-http\", \"url\": \"https://www.pathwren.workers.dev/mcp\" }",
  " }",
  "}"
 ],
 "sections": [
  {
   "heading": "MCP server",
   "text": [
    "The whole index, spoken as Model Context Protocol instead of as files. Streamable HTTP, stateless, no key, no signup, no session to keep alive. Every tool is read-only."
   ],
   "commands": [
    "# it answers right now — no account, no header ceremony",
    "curl -s https://www.pathwren.workers.dev/mcp \\",
    "  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \\",
    "  -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}' | jq -r '.result.tools[].name'"
   ],
   "tables": [],
   "links": [
    "https://modelcontextprotocol.io"
   ]
  },
  {
   "heading": "Add it to a client",
   "text": [
    "Anything that speaks MCP over HTTP. Claude Code:",
    "Clients that take a JSON config (Claude Desktop, Cursor, VS Code, Windsurf):"
   ],
   "commands": [
    "claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp",
    "{",
    " \"mcpServers\": {",
    "  \"ai-crawler-index\": { \"type\": \"streamable-http\", \"url\": \"https://www.pathwren.workers.dev/mcp\" }",
    " }",
    "}"
   ],
   "tables": [],
   "links": []
  },
  {
   "heading": "Tools",
   "text": [],
   "commands": [],
   "tables": [
    {
     "headers": [
      "Tool",
      "What it answers"
     ],
     "rows": [
      [
       "classify_user_agent",
       "Given a raw User-Agent header: which crawler is this, who runs it, what is it for, and what does blocking it cost me?"
      ],
      [
       "lookup_crawler",
       "The full record for one crawler by slug, name or robots.txt token."
      ],
      [
       "list_crawlers",
       "The 150 crawlers, filtered by category, operator, robots.txt stance or verification method. Returns the valid filter values too, so one call teaches the vocabulary."
      ],
      [
       "generate_robots_txt",
       "A ready-to-paste robots.txt for one of 8 stances, with the reasoning and the exact tokens it implies."
      ],
      [
       "is_verified_crawler_ip",
       "Is this IP inside a range the operator itself publishes? 1984 IPv4 and 1062 IPv6 prefixes, mirrored from 15 endpoints, refreshed every six hours."
      ],
      [
       "whats_changed",
       "Build time, per-source freshness, which sources changed or are failing, and the dated changelog. The cron check for anything that regenerates a config from this data."
      ]
     ]
    }
   ],
   "links": []
  },
  {
   "heading": "Resources",
   "text": [
    "Four, each pointing at the file the tools are computed from: agents.json, ua-regex.json, ip-ranges/all.json, status.json. Read them directly if you would rather do your own matching."
   ],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "What it will not do",
   "text": [
    "It has no side effects and no write path: it cannot edit your robots.txt, block anything, or fetch a URL for you. A user-agent match is a claim, not a proof — classify_user_agent says so on every answer, and is_verified_crawler_ip is the tool that turns a claim into evidence. Where an operator publishes no ranges, no tool here can verify it, and each one says which case you are in."
   ],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "Five more servers on this host",
   "text": [
    "Different jobs, no shared tool names — run all six or just the one you need. crawler-log-triage at /mcp/triage takes a whole access log and returns per-line verdicts, the impersonators, and a robots.txt or WAF ruleset for exactly what was in it. agent-discovery-doctor at /mcp/doctor turns the question around: which of the 22 discovery documents agents ask for — llms.txt, A2A agent card, owners.json, mcp.json — does a host actually serve, and who asks for each missing one. robots-policy-lint at /mcp/robots reads a robots.txt you paste and reports what it really does, by RFC 9309 rules. crawler-ip-verifier at /mcp/netcheck is the batch and prefix form of is_verified_crawler_ip: 500 addresses at a time, CIDR overlap arithmetic, and a firewall allowlist you can paste. mcp-endpoint-lint at /mcp/lint is the one aimed at MCP itself: give it another server's endpoint and it reports what the handshake negotiated, whether every advertised tool schema compiles, whether the error codes are the ones JSON-RPC requires, and what a client finds before it dials — scored out of 100.",
    "Protocol versions 2025-06-18, negotiated per call. server/discover answers for clients on 2026-07-28, initialize for everyone else. Same data as /data/agents.json and /openapi.json — pick whichever your client speaks. Listed in the official MCP Registry as dev.workers.pathwren.www/ai-crawler-index."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/mcp-triage.html",
    "/mcp-doctor.html",
    "/mcp-robots.html",
    "/mcp-netcheck.html",
    "/mcp-lint.html",
    "/data/agents.json",
    "/openapi.json",
    "https://registry.modelcontextprotocol.io/v0/servers?search=ai-crawler-index"
   ]
  }
 ],
 "machine_doors": [
  {
   "url": "https://www.pathwren.workers.dev/tools/?s=client-dossiers",
   "name": "6 keyless GET tools",
   "what": "The read-only MCP tools of this host as plain GET endpoints — no JSON-RPC, no key"
  },
  {
   "url": "https://www.pathwren.workers.dev/documents.json",
   "name": "documents.json",
   "what": "Every document here with its strong ETag and the date its bytes changed"
  },
  {
   "url": "https://www.pathwren.workers.dev/changes",
   "name": "changes",
   "what": "What moved since your cursor — poll this instead of re-downloading anything"
  },
  {
   "url": "https://www.pathwren.workers.dev/llms.txt",
   "name": "llms.txt",
   "what": "The whole map in one text file"
  },
  {
   "url": "https://www.pathwren.workers.dev/openapi.json",
   "name": "openapi.json",
   "what": "Every read endpoint, described formally"
  },
  {
   "url": "https://www.pathwren.workers.dev/.well-known/agent-card.json",
   "name": "agent card",
   "what": "A2A agent card"
  },
  {
   "url": "https://www.pathwren.workers.dev/mcp",
   "name": "mcp",
   "what": "MCP over JSON-RPC (POST)"
  },
  {
   "url": "https://www.pathwren.workers.dev/a2a",
   "name": "a2a",
   "what": "A2A (POST message/send)"
  }
 ],
 "links": [
  {
   "rel": "self",
   "href": "https://www.pathwren.workers.dev/mcp.json",
   "type": "application/json"
  },
  {
   "rel": "describes",
   "href": "https://www.pathwren.workers.dev/mcp.html",
   "type": "text/html",
   "title": "The page this document is the JSON twin of: MCP server — AI Crawler Index"
  },
  {
   "rel": "changes",
   "href": "https://www.pathwren.workers.dev/changes.json?since=111",
   "type": "application/json",
   "title": "What changed since your cursor — poll this instead of re-downloading this document",
   "cursor_param": "since",
   "head_cursor": 111,
   "min_poll_seconds": 21600,
   "how": "Read `cursor` from the response and send it back as `since`. It advances only when something really changed, so an unchanged answer is proof rather than luck — about 2.5 KB, or a 304 with no body if you send back the ETag."
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/documents.json",
   "type": "application/json",
   "title": "Every document here with its ETag and last-modified date"
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/data/agents.json",
   "type": "application/json",
   "title": "Every crawler record in one file"
  },
  {
   "rel": "service-desc",
   "href": "https://www.pathwren.workers.dev/openapi.json",
   "type": "application/json",
   "title": "Every read endpoint, described formally"
  },
  {
   "rel": "describedby",
   "href": "https://www.pathwren.workers.dev/llms.txt",
   "type": "text/plain",
   "title": "The whole map in one text file"
  }
 ]
}