curl -s https://www.pathwren.workers.dev/crawler/index.json   # this page, as JSON
curl -s https://www.pathwren.workers.dev/data/agents.json     # Every crawler record in one file

All crawlers (150)

Grouped by what the crawl is for. Machine copy: agents.json · agents.csv · user-agents.txt

AI training crawlers (27)

Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today.

AI search crawlers (21)

Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space.

User-triggered fetchers (12)

Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader.

Corpus and dataset builders (17)

Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect.

Search engines (28)

Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.

SEO and backlink crawlers (17)

Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth.

Archivers (2)

Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI one.

Link preview fetchers (4)

Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident.

Tools and frameworks (22)

Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question.