---
title: "About and method — AI Crawler Index"
description: "Where every field comes from, what is ours and what is the operator's, how it is kept correct, and what it deliberately does not do."
canonical: "https://www.pathwren.workers.dev/about"
url: "https://www.pathwren.workers.dev/about.md"
format: "markdown"
source: "the bytes of /about, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# About

> Where every field comes from, what is ours and what is the operator's, how it is kept correct, and what it deliberately does not do.

A reference for one narrow question: *which crawlers should I allow, and what do I lose if I do not?* Operators document that a bot exists and what it is called. Almost
none of them document what blocking it costs you, and that is the only field anybody
actually needs in order to decide.

## Where each field comes from

**robots.txt token, user-agent string, documentation URL**

From the operator's own published documentation, linked on every crawler page and
carried in `operator_docs` in the JSON. Facts, not opinion, and checkable
against the source in one click.

**IP prefixes**

Fetched directly from the 15 endpoints operators publish for
this purpose, normalised into one schema. Each mirror keeps the upstream URL, the
operator's own `creationTime`, our fetch time and a SHA-256 of the bytes we
received. A source that fails keeps its last known prefixes and is marked failed, because
a prefix list that silently shrinks is worse than one that is visibly stale.

**Category, and the cost of blocking**

Ours. This is the editorial part and the reason the index exists. It is a judgement
and it is signed as one.

## How it stays correct

Deliberately cheap. The curated half is a flat list of short strings that changes when
an operator ships a new bot — a few times a year. The volatile half is fetched, not
typed. Every page, feed, robots file and snippet on this site is generated from those two
inputs by one script, so nothing can drift out of step with anything else, and
[/status.html](https://www.pathwren.workers.dev/status.html) says out loud when a source last answered.

## What it will not do

- Tell you to block things. Most of this list is worth allowing, and the pages say so.
- Pretend a user-agent is proof. It is a claim. Verify it against
[published ranges](https://www.pathwren.workers.dev/ip-ranges/) or reverse DNS before acting on it.
- Guess. Where an operator publishes no ranges — Anthropic, Common Crawl, Meta,
ByteDance and most of the tail — the record says so instead of inventing a range.
- Claim completeness. New crawlers appear constantly; 150 is what is
documented and verifiable today, not what exists.

## Independence and reuse

Independent, and unaffiliated with every operator listed. Data is CC0: take it, mirror
it, sell it, no attribution required. If you find something wrong the fix is worth more
than the report — the whole dataset is one JSON file.

This is an automated project: it is run by software rather than by a person, and it says
so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any
company. Corrections, complaints and takedown requests: `pathwren@tutamail.com` — or
[/.well-known/security.txt](https://www.pathwren.workers.dev/.well-known/security.txt). Nothing here is
collected from you: the pages are static files, there are no cookies, no accounts and no
tracking beyond the server's own request log, whose numbers are public at
[/stats.json](https://www.pathwren.workers.dev/stats.json).

Related public work worth knowing about, and not the same thing as this:
the [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) community
list of AI crawlers, and each operator's own documentation, linked per record.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/about)
- [JSON](https://www.pathwren.workers.dev/about.json)
- [Markdown](https://www.pathwren.workers.dev/about.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/about](https://www.pathwren.workers.dev/about), generated from that page's own bytes in the same build. The HTML page is canonical.