{
 "name": "About and method — AI Crawler Index",
 "what": "Where every field comes from, what is ours and what is the operator's, how it is kept correct, and what it deliberately does not do.",
 "url": "https://www.pathwren.workers.dev/about.json",
 "twin_of": "https://www.pathwren.workers.dev/about.html",
 "page": {
  "path": "/about.html",
  "url": "https://www.pathwren.workers.dev/about.html",
  "type": "text/html"
 },
 "generated_at": "2026-09-03T06:29:54+00:00",
 "generated_from": "the bytes of /about.html, by surfaces/ai-crawler-index/build.py, in the same pass that wrote the page — one source, so the page and this document cannot disagree about what this host says.",
 "license": {
  "document": "CC0-1.0",
  "url": "https://creativecommons.org/publicdomain/zero/1.0/"
 },
 "access": {
  "api_key": "none",
  "account": "none",
  "rate_limit": "none",
  "cors": "*",
  "auth": "none — every document here is a public GET"
 },
 "commands": [
  "curl -s https://www.pathwren.workers.dev/about.json",
  "curl -s https://www.pathwren.workers.dev/documents.json | jq -r '.documents[] | select(.path==\"/about.json\") | .etag'"
 ],
 "sections": [
  {
   "heading": "About",
   "text": [
    "A reference for one narrow question: which crawlers should I allow, and what do I lose if I do not? Operators document that a bot exists and what it is called. Almost none of them document what blocking it costs you, and that is the only field anybody actually needs in order to decide."
   ],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "Where each field comes from",
   "text": [
    "robots.txt token, user-agent string, documentation URL",
    "From the operator's own published documentation, linked on every crawler page and carried in operator_docs in the JSON. Facts, not opinion, and checkable against the source in one click.",
    "IP prefixes",
    "Fetched directly from the 15 endpoints operators publish for this purpose, normalised into one schema. Each mirror keeps the upstream URL, the operator's own creationTime, our fetch time and a SHA-256 of the bytes we received. A source that fails keeps its last known prefixes and is marked failed, because a prefix list that silently shrinks is worse than one that is visibly stale.",
    "Category, and the cost of blocking",
    "Ours. This is the editorial part and the reason the index exists. It is a judgement and it is signed as one."
   ],
   "commands": [],
   "tables": [],
   "links": []
  },
  {
   "heading": "How it stays correct",
   "text": [
    "Deliberately cheap. The curated half is a flat list of short strings that changes when an operator ships a new bot — a few times a year. The volatile half is fetched, not typed. Every page, feed, robots file and snippet on this site is generated from those two inputs by one script, so nothing can drift out of step with anything else, and /status.html says out loud when a source last answered."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/status.html"
   ]
  },
  {
   "heading": "What it will not do",
   "text": [
    "Tell you to block things. Most of this list is worth allowing, and the pages say so.",
    "Pretend a user-agent is proof. It is a claim. Verify it against published ranges or reverse DNS before acting on it.",
    "Guess. Where an operator publishes no ranges — Anthropic, Common Crawl, Meta, ByteDance and most of the tail — the record says so instead of inventing a range.",
    "Claim completeness. New crawlers appear constantly; 150 is what is documented and verifiable today, not what exists."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/ip-ranges/"
   ]
  },
  {
   "heading": "Independence and reuse",
   "text": [
    "Independent, and unaffiliated with every operator listed. Data is CC0: take it, mirror it, sell it, no attribution required. If you find something wrong the fix is worth more than the report — the whole dataset is one JSON file.",
    "This is an automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any company. Corrections, complaints and takedown requests: pathwren@tutamail.com — or /.well-known/security.txt. Nothing here is collected from you: the pages are static files, there are no cookies, no accounts and no tracking beyond the server's own request log, whose numbers are public at /stats.json.",
    "Related public work worth knowing about, and not the same thing as this: the ai.robots.txt community list of AI crawlers, and each operator's own documentation, linked per record."
   ],
   "commands": [],
   "tables": [],
   "links": [
    "/.well-known/security.txt",
    "/stats.json",
    "https://github.com/ai-robots-txt/ai.robots.txt"
   ]
  }
 ],
 "machine_doors": [
  {
   "url": "https://www.pathwren.workers.dev/tools/?s=client-dossiers",
   "name": "6 keyless GET tools",
   "what": "The read-only MCP tools of this host as plain GET endpoints — no JSON-RPC, no key"
  },
  {
   "url": "https://www.pathwren.workers.dev/documents.json",
   "name": "documents.json",
   "what": "Every document here with its strong ETag and the date its bytes changed"
  },
  {
   "url": "https://www.pathwren.workers.dev/changes",
   "name": "changes",
   "what": "What moved since your cursor — poll this instead of re-downloading anything"
  },
  {
   "url": "https://www.pathwren.workers.dev/llms.txt",
   "name": "llms.txt",
   "what": "The whole map in one text file"
  },
  {
   "url": "https://www.pathwren.workers.dev/openapi.json",
   "name": "openapi.json",
   "what": "Every read endpoint, described formally"
  },
  {
   "url": "https://www.pathwren.workers.dev/.well-known/agent-card.json",
   "name": "agent card",
   "what": "A2A agent card"
  },
  {
   "url": "https://www.pathwren.workers.dev/mcp",
   "name": "mcp",
   "what": "MCP over JSON-RPC (POST)"
  },
  {
   "url": "https://www.pathwren.workers.dev/a2a",
   "name": "a2a",
   "what": "A2A (POST message/send)"
  }
 ],
 "links": [
  {
   "rel": "self",
   "href": "https://www.pathwren.workers.dev/about.json",
   "type": "application/json"
  },
  {
   "rel": "describes",
   "href": "https://www.pathwren.workers.dev/about.html",
   "type": "text/html",
   "title": "The page this document is the JSON twin of: About and method — AI Crawler Index"
  },
  {
   "rel": "changes",
   "href": "https://www.pathwren.workers.dev/changes.json?since=111",
   "type": "application/json",
   "title": "What changed since your cursor — poll this instead of re-downloading this document",
   "cursor_param": "since",
   "head_cursor": 111,
   "min_poll_seconds": 21600,
   "how": "Read `cursor` from the response and send it back as `since`. It advances only when something really changed, so an unchanged answer is proof rather than luck — about 2.5 KB, or a 304 with no body if you send back the ETag."
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/documents.json",
   "type": "application/json",
   "title": "Every document here with its ETag and last-modified date"
  },
  {
   "rel": "related",
   "href": "https://www.pathwren.workers.dev/data/agents.json",
   "type": "application/json",
   "title": "Every crawler record in one file"
  },
  {
   "rel": "service-desc",
   "href": "https://www.pathwren.workers.dev/openapi.json",
   "type": "application/json",
   "title": "Every read endpoint, described formally"
  },
  {
   "rel": "describedby",
   "href": "https://www.pathwren.workers.dev/llms.txt",
   "type": "text/plain",
   "title": "The whole map in one text file"
  }
 ]
}