---
title: "API reference — AI Crawler Index"
description: "Every endpoint on this host, generated from the OpenAPI document: the crawler dataset, the IP-range mirrors, the robots.txt policies, the five MCP servers and the discovery documents. No API key, no sign-up, no rate limit, CORS open, CC0."
canonical: "https://www.pathwren.workers.dev/reference"
url: "https://www.pathwren.workers.dev/reference.md"
format: "markdown"
source: "the bytes of /reference, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# API reference

> Every endpoint on this host, generated from the OpenAPI document: the crawler dataset, the IP-range mirrors, the robots.txt policies, the five MCP servers and the discovery documents. No API key, no sign-up, no rate limit, CORS open, CC0.

Every endpoint this host serves, generated from
[/openapi.json](https://www.pathwren.workers.dev/openapi.json) by the same build that writes it. No key, no
sign-up, no session, no rate limit. It works right now, from a terminal you already
have:

```bash
curl -s https://www.pathwren.workers.dev/data/agents.json | head -c 300
curl -s https://www.pathwren.workers.dev/crawler/gptbot.json
curl -s https://www.pathwren.workers.dev/robots/allow-all.txt
```

Machine copy of this page: [/reference.json](https://www.pathwren.workers.dev/reference.json) ·
formal spec: [/openapi.json](https://www.pathwren.workers.dev/openapi.json) (3.1),
[/openapi.yaml](https://www.pathwren.workers.dev/openapi.yaml), [/swagger.json](https://www.pathwren.workers.dev/swagger.json) (2.0) ·
for a model: [/llms.txt](https://www.pathwren.workers.dev/llms.txt) ·
[API keys and sign-up: there are none](https://www.pathwren.workers.dev/register).

## Before anything else

|   |   |
| --- | --- |
| Base URL | `https://www.pathwren.workers.dev` |
| Authentication | There is no key, no token, no signup and no account. Send the GET. There is nothing to [sign up](https://www.pathwren.workers.dev/register) for and no API key to obtain. |
| Methods | `GET`, `HEAD` and `OPTIONS` on every path below. Nothing here is written by a client, so nothing takes a `POST` except the JSON-RPC protocol endpoints. |
| Formats | JSON, CSV, plain text, markdown and XML, one path per shape. No content negotiation is required — the path names the format. |
| CORS | Access-Control-Allow-Origin: * on everything — callable straight from a browser. |
| Caching | Every response carries a strong ETag — sha-256 over the exact bytes, first 32 hex characters — and a Last-Modified saying when those bytes last CHANGED, not when the site was rebuilt. Both halves of RFC 9110 §13.1 work: If-None-Match and If-Modified-Since each produce a 304. You do not have to fetch a document to learn its validator: /documents.json publishes the ETag and the date for every document here, so one request tells you what moved. |
| Rate limits | None imposed by this API: every path is a static file on an edge cache, and no request is rejected for being frequent. The host's free plan allows 100,000 requests a day across the whole site, so the only real limit is that one — please fetch /data/agents.json once (about 300 KB, everything in it) rather than 150 per-crawler files, and remember the data changes at most every six hours, which /status.json tells you exactly. |
| Data licence | [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/) — public domain, no attribution required. |
| Freshness | Rebuilt every six hours; [/status.json](https://www.pathwren.workers.dev/status.json) reports when each upstream source last answered and whether it changed. |

## The data API

173 paths in the OpenAPI document, in 7 groups. Every one is a
static file on an edge cache: the same request twice is the same bytes twice, and an
`If-None-Match` gets a 304.

### bulk

The whole dataset in several shapes.

| Endpoint | What it returns |
| --- | --- |
| `GET /data/agents.csv` | The same table as CSV → `text/csv` |
| `GET /data/agents.json` | Every crawler record, plus categories and an endpoint map → `application/json` |
| `GET /data/ua-regex.json` | Pre-escaped user-agent regexes, whole-list and per category → `application/json` |

```bash
curl -s https://www.pathwren.workers.dev/data/agents.csv
curl -s https://www.pathwren.workers.dev/data/agents.json
curl -s https://www.pathwren.workers.dev/data/ua-regex.json
```

### ip-ranges

Operator-published prefixes, normalised.

| Endpoint | What it returns |
| --- | --- |
| `GET /data/ip-sources.json` | Which operators publish verifiable IP ranges, and where → `application/json` |
| `GET /ip-ranges/all.json` | Union of every operator-published prefix, grouped by source → `application/json` |
| `GET /ip-ranges/all.txt` | The same prefixes, one CIDR per line → `text/plain` |
| `GET /ip-ranges/{source}.json` | One operator's published prefix list, normalised `source`: Source slug, from /data/ip-sources.json — 15 values, e.g. `openai-gptbot`, `openai-searchbot` → `application/json` status: `200`, `404` |

```bash
curl -s https://www.pathwren.workers.dev/data/ip-sources.json
curl -s https://www.pathwren.workers.dev/ip-ranges/all.json
curl -s https://www.pathwren.workers.dev/ip-ranges/all.txt
curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json
```

### robots

Ready-made robots.txt policy files.

| Endpoint | What it returns |
| --- | --- |
| `GET /policy/{policy}.json` | A policy with its rationale and the crawlers it names `policy`: path parameter — 8 values, e.g. `allow-all`, `block-ai-training` → `application/json` |
| `GET /robots/{policy}.txt` | A ready-made robots.txt policy file `policy`: path parameter — 8 values, e.g. `allow-all`, `block-ai-training` → `text/plain` |

```bash
curl -s https://www.pathwren.workers.dev/policy/allow-all.json
curl -s https://www.pathwren.workers.dev/robots/allow-all.txt
```

### status

Freshness of the upstream sources.

| Endpoint | What it returns |
| --- | --- |
| `GET /documents.json` | The document ledger: every URL here, its ETag and its last-modified One request that tells you what to re-fetch. Every document this host publishes is listed with a strong ETag (sha-256 over the exact bytes served, first 32 hex characters) and the date those bytes last CHANGED. Compare against your copy, then GET only the paths that differ — with If-None-Match set to the etag from this file, so even a wrong guess costs a 304 and no body. HTML pages are listed with a null etag: live counters are rendered into them per request, so no validator for one could be honest. → `application/json` |
| `GET /documents.txt` | The same ledger as TSV: path, kind, bytes, etag, last-modified For the client that is a shell. Comment lines start with '#'; every other line is five tab-separated fields. A '-' means no validator is published for that document. → `text/plain` |
| `GET /feed.json` | JSON Feed 1.1 of what changed → `application/feed+json` |
| `GET /status.json` | Freshness and health of every upstream IP-range endpoint → `application/json` |

```bash
curl -s https://www.pathwren.workers.dev/documents.json
curl -s https://www.pathwren.workers.dev/documents.txt
curl -s https://www.pathwren.workers.dev/feed.json
curl -s https://www.pathwren.workers.dev/status.json
```

### changes

What changed since your last read, with a cursor to send back. The cheap way to stay current without re-downloading anything.

| Endpoint | What it returns |
| --- | --- |
| `GET /changes.json` | What changed since your cursor A since-cursor feed of every real change to this index: operator IP-range lists that gained or lost prefixes, upstreams that started or stopped answering, and crawler records added, edited or withdrawn.

Read `cursor` from the response and send it back as `since` next time. The cursor is a monotonic integer that only advances when something actually changed, so an unchanged answer is proof of nothing new — not a coincidence of timing. An empty page is about 2.5 KB; sending the response's ETag back as `If-None-Match` makes it a 304 with no body at all.

The 15 upstream endpoints are re-fetched every six hours, so polling faster than that returns the same cursor. Nothing is rate limited; the request is simply not worth your budget.

`stale_sources` is repeated in EVERY response, including empty ones: when an upstream fails, this host keeps serving its last good prefixes rather than an empty list, and a client that polls daily should not have to have been listening at the exact minute it broke to find out. `since`: A cursor from a previous response, or an ISO-8601 timestamp. Omit it to read from the oldest retained event. An unparseable value is a 400, never a silently ignored parameter. `limit`: Events per page, 1–400. Default 100. Follow `next` when `has_more` is true. → `application/json` status: `200`, `304`, `400` |

```bash
curl -s https://www.pathwren.workers.dev/changes.json
```

### tools

The read-only tools this host runs over MCP, as keyless GET endpoints. Same implementation, no JSON-RPC session required.

| Endpoint | What it returns |
| --- | --- |
| `GET /tools/ai-access` | Which AI crawlers does this file actually stop? Evaluate a pasted robots.txt against every AI crawler in this index and get the two lists that matter — blocked and allowed, per operator and category — plus the tokens in your file that match no known crawler (a typo blocks nothing), and the separation between crawlers that document obedience and the ones observed ignoring robots.txt, which need an IP or WAF rule instead.

No account, no key, no handshake. Called with no parameters at all it answers 200 with its own specification rather than 400, so walking /tools/ is free. This endpoint holds no copy of the logic: it calls the `audit_ai_access` MCP tool at https://www.pathwren.workers.dev/mcp/robots in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/ai-access.html · machine spec: https://www.pathwren.workers.dev/tools/ai-access.json `robots_txt`: The contents of the robots.txt file, URL-encoded, or the raw POST body. `path`: Path to test the verdict at. Defaults to /. `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `POST /tools/ai-access` | Which AI crawlers does this file actually stop? (file as the raw body) Identical to the GET, with the `robots_txt` argument taken from the raw request body instead of the query string — for files too large or too awkward to URL-encode. Still keyless. → `application/json` status: `200`, `400` |
| `GET /tools/classify-ua` | What is this user-agent? Identify the crawler behind a raw User-Agent header: operator, category, robots.txt token and stance, how it can be verified, and what blocking it costs. An unmatched string is reported as unmatched, with a labelled substring heuristic and never an identification.

No account, no key, no handshake. Called with no parameters at all it answers 200 with its own specification rather than 400, so walking /tools/ is free. This endpoint holds no copy of the logic: it calls the `classify_user_agent` MCP tool at https://www.pathwren.workers.dev/mcp in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/classify-ua.html · machine spec: https://www.pathwren.workers.dev/tools/classify-ua.json `ua`: The raw User-Agent header value, URL-encoded. `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `GET /tools/example` | A worked example, run for real — no arguments Takes no arguments at all. It runs this host's flagship tool (classify_user_agent) on a canned input from our own published log — the busiest client we have seen — and returns exactly the answer a real call returns, not a mock. Use it to see the shape of an answer before deciding what to send.

No account, no key, no handshake. This endpoint takes no parameters at all: a bare GET is the whole call and returns the real answer. Add ?spec=1 for the specification instead. It is the one call you can make here before you know anything about this host. This endpoint holds no copy of the logic: it calls the `example` MCP tool at https://www.pathwren.workers.dev/mcp in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/example.html · machine spec: https://www.pathwren.workers.dev/tools/example.json `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `GET /tools/robots-allowed` | Would this crawler fetch this path? Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309 — longest token match for the group, longest pattern for the rule, Allow breaking a tie, * and $ supported — and get allowed/disallowed per path with the exact line that decided it, plus a flag on the cases where a merge-groups parser and a first-group-wins parser would disagree.

No account, no key, no handshake. Called with no parameters at all it answers 200 with its own specification rather than 400, so walking /tools/ is free. This endpoint holds no copy of the logic: it calls the `check_path_allowed` MCP tool at https://www.pathwren.workers.dev/mcp/robots in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/robots-allowed.html · machine spec: https://www.pathwren.workers.dev/tools/robots-allowed.json `robots_txt`: The contents of the robots.txt file, URL-encoded, or the raw POST body. `ua`: The crawler's product token — GPTBot, Googlebot, ClaudeBot — not a whole User-Agent header. `path`: Path to test. Repeat for several, or send a comma separated list. Defaults to /. Max 200. Repeatable. `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `POST /tools/robots-allowed` | Would this crawler fetch this path? (file as the raw body) Identical to the GET, with the `robots_txt` argument taken from the raw request body instead of the query string — for files too large or too awkward to URL-encode. Still keyless. → `application/json` status: `200`, `400` |
| `GET /tools/robots-lint` | Lint a robots.txt Paste a robots.txt and get every fault that makes it do something other than what it looks like: misspelled directives, a whole User-Agent string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, a byte-order mark. Each finding carries the line number and the fix.

No account, no key, no handshake. Called with no parameters at all it answers 200 with its own specification rather than 400, so walking /tools/ is free. This endpoint holds no copy of the logic: it calls the `lint_robots_txt` MCP tool at https://www.pathwren.workers.dev/mcp/robots in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/robots-lint.html · machine spec: https://www.pathwren.workers.dev/tools/robots-lint.json `robots_txt`: The contents of the robots.txt file, URL-encoded. Or POST the file as the raw request body, which is what a file larger than a URL wants. `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `POST /tools/robots-lint` | Lint a robots.txt (file as the raw body) Identical to the GET, with the `robots_txt` argument taken from the raw request body instead of the query string — for files too large or too awkward to URL-encode. Still keyless. → `application/json` status: `200`, `400` |
| `GET /tools/verification-methods` | Can this crawler be verified at all, and how? Before you read anything into a miss from /tools/verify-crawler: which crawlers can be verified by published prefix, which document forward-confirmed reverse DNS instead (with the exact two dig commands to run), and which publish no verification method at all — with the mirror's own counts and fetch dates.

No account, no key, no handshake. Called with no parameters at all it answers 200 with its own specification rather than 400, so walking /tools/ is free. This endpoint holds no copy of the logic: it calls the `verification_methods` MCP tool at https://www.pathwren.workers.dev/mcp/netcheck in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/verification-methods.html · machine spec: https://www.pathwren.workers.dev/tools/verification-methods.json `crawler`: Crawler slug, name, operator or user-agent substring. Omit for the whole tally. `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `GET /tools/verify-crawler` | Does this address really belong to the crawler it claims to be? Check one or more IP addresses against every crawler prefix the operators themselves publish (Google, OpenAI, Anthropic, Microsoft, Apple, Perplexity, Amazon and the rest, remirrored every six hours), and — when you name the crawler the client claimed to be — say whether the range that contains it belongs to that operator or to a different one.

No account, no key, no handshake. Called with no parameters at all it answers 200 with its own specification rather than 400, so walking /tools/ is free. This endpoint holds no copy of the logic: it calls the `verify_batch` MCP tool at https://www.pathwren.workers.dev/mcp/netcheck in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/verify-crawler.html · machine spec: https://www.pathwren.workers.dev/tools/verify-crawler.json `ip`: IPv4 or IPv6 address. Repeat the parameter, or send a comma/space separated list. Up to 500 per call. Repeatable. `ua`: The crawler the client claimed to be — a product token like GPTBot or a whole User-Agent header. Applied to every address in the call. `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |
| `GET /tools/whoami` | Who is calling? — the one endpoint that needs nothing from you Takes no arguments at all. It answers about the request you just made: the user-agent you sent, the address you came from, whether the crawler index recognises you, whether any operator publishes a prefix containing your address, the class this host's own instrument books you as, and whether we have seen you here before and what you fetched. Every fact comes from your own request headers or from a file this host already publishes; nothing is fetched and nothing about you is invented.

No account, no key, no handshake. This endpoint takes no parameters at all: a bare GET is the whole call and returns the real answer. Add ?spec=1 for the specification instead. It is the one call you can make here before you know anything about this host. This endpoint holds no copy of the logic: it calls the `whoami` MCP tool at https://www.pathwren.workers.dev/mcp in-process, through that server's own JSON-RPC envelope. One implementation, two doors.

Docs: https://www.pathwren.workers.dev/tools/whoami.html · machine spec: https://www.pathwren.workers.dev/tools/whoami.json `s`: Attribution tag. Not used to compute the answer: it tells this host which of its surfaces sent you, which is how it decides what to keep building. Leave the default in place if you have no reason not to. → `application/json` status: `200`, `304`, `400`, `405` |

```bash
curl -s https://www.pathwren.workers.dev/tools/ai-access
curl -s https://www.pathwren.workers.dev/tools/ai-access
curl -s https://www.pathwren.workers.dev/tools/classify-ua
curl -s https://www.pathwren.workers.dev/tools/example
curl -s https://www.pathwren.workers.dev/tools/robots-allowed
curl -s https://www.pathwren.workers.dev/tools/robots-allowed
curl -s https://www.pathwren.workers.dev/tools/robots-lint
curl -s https://www.pathwren.workers.dev/tools/robots-lint
curl -s https://www.pathwren.workers.dev/tools/verification-methods
curl -s https://www.pathwren.workers.dev/tools/verify-crawler
curl -s https://www.pathwren.workers.dev/tools/whoami
```

### One file per crawler

The 151 paths of the form `/crawler/<slug>.json` are described
individually in [/openapi.json](https://www.pathwren.workers.dev/openapi.json) and are the same records as
[/data/agents.json](https://www.pathwren.workers.dev/data/agents.json), one file each. The same record is served
three ways — `.html` for a person, `.json` for a program,
`.md` for a model. Slugs:

[gptbot](https://www.pathwren.workers.dev/crawler/gptbot.html), [oai-searchbot](https://www.pathwren.workers.dev/crawler/oai-searchbot.html), [chatgpt-user](https://www.pathwren.workers.dev/crawler/chatgpt-user.html), [claudebot](https://www.pathwren.workers.dev/crawler/claudebot.html), [claude-searchbot](https://www.pathwren.workers.dev/crawler/claude-searchbot.html), [claude-user](https://www.pathwren.workers.dev/crawler/claude-user.html), [anthropic-ai](https://www.pathwren.workers.dev/crawler/anthropic-ai.html), [claude-web](https://www.pathwren.workers.dev/crawler/claude-web.html), [google-extended](https://www.pathwren.workers.dev/crawler/google-extended.html), [googlebot](https://www.pathwren.workers.dev/crawler/googlebot.html), [googleother](https://www.pathwren.workers.dev/crawler/googleother.html), [google-cloudvertexbot](https://www.pathwren.workers.dev/crawler/google-cloudvertexbot.html), [google-inspectiontool](https://www.pathwren.workers.dev/crawler/google-inspectiontool.html), [googlebot-image](https://www.pathwren.workers.dev/crawler/googlebot-image.html), [googlebot-news](https://www.pathwren.workers.dev/crawler/googlebot-news.html), [storebot-google](https://www.pathwren.workers.dev/crawler/storebot-google.html), [bingbot](https://www.pathwren.workers.dev/crawler/bingbot.html), [applebot](https://www.pathwren.workers.dev/crawler/applebot.html), [applebot-extended](https://www.pathwren.workers.dev/crawler/applebot-extended.html), [perplexitybot](https://www.pathwren.workers.dev/crawler/perplexitybot.html), [perplexity-user](https://www.pathwren.workers.dev/crawler/perplexity-user.html), [ccbot](https://www.pathwren.workers.dev/crawler/ccbot.html), [bytespider](https://www.pathwren.workers.dev/crawler/bytespider.html), [tiktokspider](https://www.pathwren.workers.dev/crawler/tiktokspider.html), [meta-externalagent](https://www.pathwren.workers.dev/crawler/meta-externalagent.html), [meta-externalfetcher](https://www.pathwren.workers.dev/crawler/meta-externalfetcher.html), [facebookexternalhit](https://www.pathwren.workers.dev/crawler/facebookexternalhit.html), [facebookbot](https://www.pathwren.workers.dev/crawler/facebookbot.html), [amazonbot](https://www.pathwren.workers.dev/crawler/amazonbot.html), [duckassistbot](https://www.pathwren.workers.dev/crawler/duckassistbot.html), [duckduckbot](https://www.pathwren.workers.dev/crawler/duckduckbot.html), [ai2bot](https://www.pathwren.workers.dev/crawler/ai2bot.html), [ai2bot-dolma](https://www.pathwren.workers.dev/crawler/ai2bot-dolma.html), [cohere-ai](https://www.pathwren.workers.dev/crawler/cohere-ai.html), [cohere-training-data-crawler](https://www.pathwren.workers.dev/crawler/cohere-training-data-crawler.html), [mistralai-user](https://www.pathwren.workers.dev/crawler/mistralai-user.html), [youbot](https://www.pathwren.workers.dev/crawler/youbot.html), [diffbot](https://www.pathwren.workers.dev/crawler/diffbot.html), [omgilibot](https://www.pathwren.workers.dev/crawler/omgilibot.html), [omgili](https://www.pathwren.workers.dev/crawler/omgili.html), [webzio-extended](https://www.pathwren.workers.dev/crawler/webzio-extended.html), [imagesiftbot](https://www.pathwren.workers.dev/crawler/imagesiftbot.html), [timpibot](https://www.pathwren.workers.dev/crawler/timpibot.html), [semrushbot](https://www.pathwren.workers.dev/crawler/semrushbot.html), [semrushbot-ocob](https://www.pathwren.workers.dev/crawler/semrushbot-ocob.html), [ahrefsbot](https://www.pathwren.workers.dev/crawler/ahrefsbot.html), [archive-org-bot](https://www.pathwren.workers.dev/crawler/archive-org-bot.html), [ia-archiver](https://www.pathwren.workers.dev/crawler/ia-archiver.html), [yandexbot](https://www.pathwren.workers.dev/crawler/yandexbot.html), [baiduspider](https://www.pathwren.workers.dev/crawler/baiduspider.html), [seznambot](https://www.pathwren.workers.dev/crawler/seznambot.html), [yeti](https://www.pathwren.workers.dev/crawler/yeti.html), [petalbot](https://www.pathwren.workers.dev/crawler/petalbot.html), [firecrawlagent](https://www.pathwren.workers.dev/crawler/firecrawlagent.html), [scrapy](https://www.pathwren.workers.dev/crawler/scrapy.html), [img2dataset](https://www.pathwren.workers.dev/crawler/img2dataset.html), [googlebot-video](https://www.pathwren.workers.dev/crawler/googlebot-video.html), [googleother-image](https://www.pathwren.workers.dev/crawler/googleother-image.html), [googleother-video](https://www.pathwren.workers.dev/crawler/googleother-video.html), [apis-google](https://www.pathwren.workers.dev/crawler/apis-google.html), [adsbot-google](https://www.pathwren.workers.dev/crawler/adsbot-google.html), [adsbot-google-mobile](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile.html), [adsbot-google-mobile-apps](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile-apps.html), [mediapartners-google](https://www.pathwren.workers.dev/crawler/mediapartners-google.html), [google-safety](https://www.pathwren.workers.dev/crawler/google-safety.html), [feedfetcher-google](https://www.pathwren.workers.dev/crawler/feedfetcher-google.html), [google-read-aloud](https://www.pathwren.workers.dev/crawler/google-read-aloud.html), [google-site-verification](https://www.pathwren.workers.dev/crawler/google-site-verification.html), [google-cws](https://www.pathwren.workers.dev/crawler/google-cws.html), [google-pinpoint](https://www.pathwren.workers.dev/crawler/google-pinpoint.html), [googleproducer](https://www.pathwren.workers.dev/crawler/googleproducer.html), [googlemessages](https://www.pathwren.workers.dev/crawler/googlemessages.html), [google-gemininotebook](https://www.pathwren.workers.dev/crawler/google-gemininotebook.html), [google-agent](https://www.pathwren.workers.dev/crawler/google-agent.html), [yandeximages](https://www.pathwren.workers.dev/crawler/yandeximages.html), [yandexvideo](https://www.pathwren.workers.dev/crawler/yandexvideo.html), [yandexmedia](https://www.pathwren.workers.dev/crawler/yandexmedia.html), [yandexblogs](https://www.pathwren.workers.dev/crawler/yandexblogs.html), [yandexmarket](https://www.pathwren.workers.dev/crawler/yandexmarket.html), [yandexwebmaster](https://www.pathwren.workers.dev/crawler/yandexwebmaster.html), [yandexmobilebot](https://www.pathwren.workers.dev/crawler/yandexmobilebot.html), [yandexfavicons](https://www.pathwren.workers.dev/crawler/yandexfavicons.html), [yandexcalendar](https://www.pathwren.workers.dev/crawler/yandexcalendar.html), [yandexdirect](https://www.pathwren.workers.dev/crawler/yandexdirect.html), [yandexmetrika](https://www.pathwren.workers.dev/crawler/yandexmetrika.html), [yandexrenderresourcesbot](https://www.pathwren.workers.dev/crawler/yandexrenderresourcesbot.html), [yandexscreenshotbot](https://www.pathwren.workers.dev/crawler/yandexscreenshotbot.html), [yandexadditional](https://www.pathwren.workers.dev/crawler/yandexadditional.html), [yandexadditionalbot](https://www.pathwren.workers.dev/crawler/yandexadditionalbot.html), [yandexcombot](https://www.pathwren.workers.dev/crawler/yandexcombot.html), [siteauditbot](https://www.pathwren.workers.dev/crawler/siteauditbot.html), [semrushbot-ba](https://www.pathwren.workers.dev/crawler/semrushbot-ba.html), [semrushbot-si](https://www.pathwren.workers.dev/crawler/semrushbot-si.html), [semrushbot-swa](https://www.pathwren.workers.dev/crawler/semrushbot-swa.html), [splitsignalbot](https://www.pathwren.workers.dev/crawler/splitsignalbot.html), [semrushbot-ft](https://www.pathwren.workers.dev/crawler/semrushbot-ft.html), [semrushbot-esi](https://www.pathwren.workers.dev/crawler/semrushbot-esi.html), [ahrefssiteaudit](https://www.pathwren.workers.dev/crawler/ahrefssiteaudit.html), [mj12bot](https://www.pathwren.workers.dev/crawler/mj12bot.html), [dotbot](https://www.pathwren.workers.dev/crawler/dotbot.html), [rogerbot](https://www.pathwren.workers.dev/crawler/rogerbot.html), [dataforseobot](https://www.pathwren.workers.dev/crawler/dataforseobot.html), [serpstatbot](https://www.pathwren.workers.dev/crawler/serpstatbot.html), [barkrowler](https://www.pathwren.workers.dev/crawler/barkrowler.html), [screaming-frog-seo-spider](https://www.pathwren.workers.dev/crawler/screaming-frog-seo-spider.html), [seokicks](https://www.pathwren.workers.dev/crawler/seokicks.html), [mojeekbot](https://www.pathwren.workers.dev/crawler/mojeekbot.html), [kagibot](https://www.pathwren.workers.dev/crawler/kagibot.html), [qwantbot](https://www.pathwren.workers.dev/crawler/qwantbot.html), [qwantbot-news](https://www.pathwren.workers.dev/crawler/qwantbot-news.html), [slackbot-linkexpanding](https://www.pathwren.workers.dev/crawler/slackbot-linkexpanding.html), [slackbot](https://www.pathwren.workers.dev/crawler/slackbot.html), [pinterestbot](https://www.pathwren.workers.dev/crawler/pinterestbot.html), [bedrockbot](https://www.pathwren.workers.dev/crawler/bedrockbot.html), [cloudflare-autorag](https://www.pathwren.workers.dev/crawler/cloudflare-autorag.html), [exasearchbot](https://www.pathwren.workers.dev/crawler/exasearchbot.html), [shapbot](https://www.pathwren.workers.dev/crawler/shapbot.html), [terracotta](https://www.pathwren.workers.dev/crawler/terracotta.html), [crawlspace](https://www.pathwren.workers.dev/crawler/crawlspace.html), [panscient](https://www.pathwren.workers.dev/crawler/panscient.html), [sbintuitionsbot](https://www.pathwren.workers.dev/crawler/sbintuitionsbot.html), [icc-crawler](https://www.pathwren.workers.dev/crawler/icc-crawler.html), [cotoyogi](https://www.pathwren.workers.dev/crawler/cotoyogi.html), [isscyberriskcrawler](https://www.pathwren.workers.dev/crawler/isscyberriskcrawler.html), [sidetrade-indexer-bot](https://www.pathwren.workers.dev/crawler/sidetrade-indexer-bot.html), [yak](https://www.pathwren.workers.dev/crawler/yak.html), [atlassian-bot](https://www.pathwren.workers.dev/crawler/atlassian-bot.html), [klaviyoaibot](https://www.pathwren.workers.dev/crawler/klaviyoaibot.html), [quillbot](https://www.pathwren.workers.dev/crawler/quillbot.html), [phindbot](https://www.pathwren.workers.dev/crawler/phindbot.html), [andibot](https://www.pathwren.workers.dev/crawler/andibot.html), [anomura](https://www.pathwren.workers.dev/crawler/anomura.html), [aiwebindex](https://www.pathwren.workers.dev/crawler/aiwebindex.html), [factset-spyderbot](https://www.pathwren.workers.dev/crawler/factset-spyderbot.html), [poseidon-research-crawler](https://www.pathwren.workers.dev/crawler/poseidon-research-crawler.html), [qualifiedbot](https://www.pathwren.workers.dev/crawler/qualifiedbot.html), [reflectionbot](https://www.pathwren.workers.dev/crawler/reflectionbot.html), [thinkbot](https://www.pathwren.workers.dev/crawler/thinkbot.html), [aihitbot](https://www.pathwren.workers.dev/crawler/aihitbot.html), [linguee-bot](https://www.pathwren.workers.dev/crawler/linguee-bot.html), [lightpanda](https://www.pathwren.workers.dev/crawler/lightpanda.html), [laiondownloader](https://www.pathwren.workers.dev/crawler/laiondownloader.html), [velenpublicwebcrawler](https://www.pathwren.workers.dev/crawler/velenpublicwebcrawler.html), [awariosmartbot](https://www.pathwren.workers.dev/crawler/awariosmartbot.html), [awariorssbot](https://www.pathwren.workers.dev/crawler/awariorssbot.html), [echoboxbot](https://www.pathwren.workers.dev/crawler/echoboxbot.html), [meta-webindexer](https://www.pathwren.workers.dev/crawler/meta-webindexer.html), [chatgpt-agent](https://www.pathwren.workers.dev/crawler/chatgpt-agent.html), [wpbot](https://www.pathwren.workers.dev/crawler/wpbot.html), [crawl4ai](https://www.pathwren.workers.dev/crawler/crawl4ai.html)

## Protocol endpoints

These are not in the OpenAPI document above, because they are not REST: they take a
JSON-RPC 2.0 `POST` and speak MCP over Streamable HTTP. Same data, same absence of
a key. Connect an agent straight to the URL, or call `tools/list` with curl.

| Endpoint | What it is |
| --- | --- |
| `POST /mcp` | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: [/mcp.html](https://www.pathwren.workers.dev/mcp.html) |
| `POST /mcp/doctor` | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: [/mcp-doctor.html](https://www.pathwren.workers.dev/mcp-doctor.html) |
| `POST /mcp/lint` | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: [/mcp-lint.html](https://www.pathwren.workers.dev/mcp-lint.html) |
| `POST /mcp/netcheck` | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: [/mcp-netcheck.html](https://www.pathwren.workers.dev/mcp-netcheck.html) |
| `POST /mcp/robots` | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: [/mcp-robots.html](https://www.pathwren.workers.dev/mcp-robots.html) |
| `POST /mcp/triage` | JSON-RPC 2.0 over Streamable HTTP. Tools, docs and examples: [/mcp-triage.html](https://www.pathwren.workers.dev/mcp-triage.html) |

```bash
curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/doctor -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/lint -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/netcheck -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/robots -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://www.pathwren.workers.dev/mcp/triage -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

The same skills as an A2A agent: [/a2a.html](https://www.pathwren.workers.dev/a2a.html),
agent card at [/.well-known/agent-card.json](https://www.pathwren.workers.dev/.well-known/agent-card.json).

## Documents about this host

What a crawler, a directory or a model reads to find out what this is, without asking a
person.

| Document | What it says |
| --- | --- |
| `/llms.txt` | The map of this site for a model: what is here and where. |
| `/llms-full.txt` | The whole index as one markdown file. |
| `/openapi.json` | OpenAPI 3.1 — every endpoint above, described formally. |
| `/openapi.yaml` | The same document as YAML, from the same object. |
| `/swagger.json` | Mechanical Swagger 2.0 conversion, served only while it stays faithful. |
| `/apis.json` | APIs.json collection entry for this host. |
| `/sitemap.xml` | Every page. |
| `/index.json` | The host in one object, with the machine entry points listed. |
| `/.well-known/api-catalog` | RFC 9727 service catalog. |
| `/.well-known/api-onboarding` | How access works, machine-readable. |
| `/.well-known/security.txt` | RFC 9116 contact and policy. |
| `/security.json` | What this host runs, and why each probed path is a genuine 404. |
| `/terms.json` | Terms, machine-readable. |
| `/privacy.json` | What is logged and what is not. |
| `/feed.json` | JSON Feed of what changed. |
| `/feed.xml` | The same as RSS. |
| `/register` | Access, keys and sign-up: there are none. |

## The one endpoint that takes a credential

Exactly one path on this host is not open: `/hits.json`, the raw export of this
host's own request log. Without the key it answers `401`, and there is no form to
request one. That is deliberate rather than an oversight — those rows are other visitors'
requests, and the salted address hashes in them are nobody else's business. The aggregate of
the same data is public, unauthenticated and CORS-open:

```bash
curl -s https://www.pathwren.workers.dev/stats.json
```

What is logged and what is not: [/privacy.html](https://www.pathwren.workers.dev/privacy.html) ·
[/security.html](https://www.pathwren.workers.dev/security.html) · [/terms.html](https://www.pathwren.workers.dev/terms.html).

## Errors

| Status | When |
| --- | --- |
| `200` | The document. Every response carries an `ETag`. |
| `304` | Your `If-None-Match` matched. Nothing changed. |
| `401` | `/hits.json` without the export key. The only one. |
| `404` | No such crawler, policy or source — and, on a handful of paths, a deliberate 404 whose reasons are published in [/security.json](https://www.pathwren.workers.dev/security.json) rather than left to guesswork. |
| `405` | A method this path does not answer, always with an `Allow` header naming the ones it does. |

## Where this page is served

The same document answers at `/reference`,
`/docs`, `/documentation`, `/api`, `/api/docs` — every alias carrying
`Link: rel="canonical"` back to `/reference` — and as JSON at
[/reference.json](https://www.pathwren.workers.dev/reference.json), or at any of those paths with
`Accept: application/json`. It is plain generated HTML with no JavaScript on it at
all: everything above is in the bytes curl returns, which is the property a
“crawler-friendly docs platform” is a proxy for.

## Why this page exists

A directory scanner calling itself
`AgentDisco/1.0 (+https://agentdisco.io/bot)` asked this host for
`/docs`, `/documentation`, `/api`, `/api/docs`,
`/reference` and `/developers` — six requests each, 2026-09-01 02:28:34Z
to 17:02:43Z — and got a 404 on thirty of the thirty-six. Its published check
(`docs.platform`, weight 6) probes exactly those six paths and grades a host on
whether its documentation can be found at one of them. The API was real and described the whole
time; the description was just not at any address the question is asked from. That is worth a
page rather than a silence, and the page is generated from the specification rather than typed,
so it cannot drift away from the API it documents. An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/reference)
- [JSON](https://www.pathwren.workers.dev/reference.json)
- [Markdown](https://www.pathwren.workers.dev/reference.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/reference](https://www.pathwren.workers.dev/reference), generated from that page's own bytes in the same build. The HTML page is canonical.