---
title: "API — AI Crawler Index"
description: "Every read endpoint, with a curl. OpenAPI 3.1 at /openapi.json. No key, no rate limit, CORS open."
canonical: "https://www.pathwren.workers.dev/api.html"
url: "https://www.pathwren.workers.dev/api.md"
format: "markdown"
source: "the bytes of /api.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# API

> Every read endpoint, with a curl. OpenAPI 3.1 at /openapi.json. No key, no rate limit, CORS open.

Everything is a static file served with `Access-Control-Allow-Origin: *`.
There is no key, no quota and no state, so the description below is complete and will
not surprise you.

[API keys and sign-up: there are none — how to authenticate, and the one endpoint that does take a credential](https://www.pathwren.workers.dev/register)

The complete reference, generated from the OpenAPI document rather than typed —
every path, every parameter, every curl: [/reference](https://www.pathwren.workers.dev/reference)
(also at `/docs`, `/documentation`, `/api`,
`/api/docs`; as data at [/reference.json](https://www.pathwren.workers.dev/reference.json)).
The page below is the short version.

```bash
curl -s https://www.pathwren.workers.dev/openapi.json | jq '.paths | keys | length'
```

## Read endpoints

| Path | Returns |
| --- | --- |
| `/data/agents.json` | all 150 crawler records + category definitions |
| `/crawler/<slug>.json` | one crawler; slugs from agents.json |
| `/operator/<slug>.json` | one of 74 operators and its crawlers |
| `/category/<slug>.json` | one of 9 categories |
| `/policy/<slug>.json` | a robots.txt policy, its rationale, and the file |
| `/robots/<slug>.txt` | the robots.txt file itself, ready to append |
| `/ip-ranges/all.json` | union of published prefixes, grouped by source |
| `/ip-ranges/<source>.json` | one operator endpoint, normalised, with provenance |
| `/data/ua-regex.json` | pre-escaped regexes, whole-list and per category |
| `/status.json` | upstream freshness |
| `/feed.json`, `/feed.xml` | what changed |

## Worked examples

```text
# every token that only affects training, as a robots.txt block
curl -s https://www.pathwren.workers.dev/data/agents.json \
  | jq -r '.crawlers[] | select(.category=="ai-training") | "User-agent: \(.robots_token)\nDisallow: /\n"'

# is this request really GPTBot?
curl -s https://www.pathwren.workers.dev/ip-ranges/openai-gptbot.json | jq -r '.ipv4[]' \
  | while read c; do python3 -c "import ipaddress,sys;print(sys.argv[1]) if ipaddress.ip_address('$1') in ipaddress.ip_network(sys.argv[1]) else None" "$c"; done

# what does blocking this cost me?
curl -s https://www.pathwren.workers.dev/crawler/oai-searchbot.json | jq -r .cost_of_blocking
```

Machine description: [/openapi.json](https://www.pathwren.workers.dev/openapi.json) ·
[/openapi.yaml](https://www.pathwren.workers.dev/openapi.yaml) ·
catalogue: [/.well-known/api-catalog](https://www.pathwren.workers.dev/.well-known/api-catalog) ·
APIs.json: [/apis.json](https://www.pathwren.workers.dev/apis.json) ·
getting started: [/.well-known/api-onboarding](https://www.pathwren.workers.dev/.well-known/api-onboarding) ·
prose for models: [/llms.txt](https://www.pathwren.workers.dev/llms.txt),
[/llms-full.txt](https://www.pathwren.workers.dev/llms-full.txt)

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/api.html)
- [JSON](https://www.pathwren.workers.dev/api.json)
- [Markdown](https://www.pathwren.workers.dev/api.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/api.html](https://www.pathwren.workers.dev/api.html), generated from that page's own bytes in the same build. The HTML page is canonical.