curl -s https://www.pathwren.workers.dev/policy/block-datasets.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Refuse the crawlers whose output is a dataset other people train on: Common Crawl, AI2, Webz.io, Diffbot, ImagesiftBot.
curl -s https://www.pathwren.workers.dev/robots/block-datasets.txt >> robots.txt
These are the highest-leverage blocks per line, because one crawl becomes many downstream training runs. It is also the block with the longest delay before it has any effect, and no effect at all on archives already published.
AI2Bot · Ai2Bot-Dolma · aiHitBot · AwarioRssBot · AwarioSmartBot · CCBot · Diffbot · EchoboxBot · ImagesiftBot · img2dataset · LAIONDownloader · omgili · omgilibot · Panscient · Thinkbot · VelenPublicWebCrawler · YaK
/robots/block-datasets.txt · json
# AI Crawler Index — policy: block-datasets # Block corpus and dataset builders # Refuse the crawlers whose output is a dataset other people train on: Common Crawl, AI2, Webz.io, Diffbot, ImagesiftBot. # Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/block-datasets.html # 17 crawlers named. Paste into robots.txt at your document root. User-agent: AI2Bot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: aiHitBot Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: CCBot Disallow: / User-agent: Diffbot Disallow: / User-agent: EchoboxBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: img2dataset Disallow: / User-agent: LAIONDownloader # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: panscient.com Disallow: / User-agent: Thinkbot # compliance disputed; enforce at the edge Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: YaK Disallow: / User-agent: * Allow: / Sitemap: https://www.pathwren.workers.dev/sitemap.xml