curl -s https://www.pathwren.workers.dev/policy/block-disputed.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead.
curl -s https://www.pathwren.workers.dev/robots/block-disputed.txt >> robots.txt
A robots.txt rule is a request. For the operators in this file the request is documented as unreliable or explicitly not applicable, so the honest use of this file is as a record of intent that sits alongside a real block by user-agent or by IP at your CDN.
Bytespider · FeedFetcher-Google · Google-Agent · Google-CWS · Google-GeminiNotebook · Google-Pinpoint · Google-Read-Aloud · Google-Safety · Google-Site-Verification · GoogleMessages · GoogleProducer · ISSCyberRiskCrawler · LAIONDownloader · Linguee Bot · Perplexity-User · Thinkbot · TikTokSpider · YandexMetrika
/robots/block-disputed.txt · json
# AI Crawler Index — policy: block-disputed # Block the crawlers with disputed robots compliance # The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead. # Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/block-disputed.html # 18 crawlers named. Paste into robots.txt at your document root. User-agent: Bytespider # compliance disputed; enforce at the edge Disallow: / User-agent: FeedFetcher-Google # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Agent # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-CWS # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-GeminiNotebook # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Pinpoint # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Read-Aloud # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Safety # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Site-Verification # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: GoogleMessages # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: GoogleProducer # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: ISSCyberRiskCrawler # compliance disputed; enforce at the edge Disallow: / User-agent: LAIONDownloader # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Linguee Bot # compliance disputed; enforce at the edge Disallow: / User-agent: Perplexity-User # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Thinkbot # compliance disputed; enforce at the edge Disallow: / User-agent: TikTokSpider # compliance disputed; enforce at the edge Disallow: / User-agent: YandexMetrika # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: * Allow: / Sitemap: https://www.pathwren.workers.dev/sitemap.xml