# AI Crawler Index — policy: block-disputed # Block the crawlers with disputed robots compliance # The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead. # Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/block-disputed.html # 18 crawlers named. Paste into robots.txt at your document root. User-agent: Bytespider # compliance disputed; enforce at the edge Disallow: / User-agent: FeedFetcher-Google # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Agent # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-CWS # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-GeminiNotebook # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Pinpoint # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Read-Aloud # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Safety # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Google-Site-Verification # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: GoogleMessages # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: GoogleProducer # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: ISSCyberRiskCrawler # compliance disputed; enforce at the edge Disallow: / User-agent: LAIONDownloader # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Linguee Bot # compliance disputed; enforce at the edge Disallow: / User-agent: Perplexity-User # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: Thinkbot # compliance disputed; enforce at the edge Disallow: / User-agent: TikTokSpider # compliance disputed; enforce at the edge Disallow: / User-agent: YandexMetrika # operator states robots.txt does not apply; enforce at the edge Disallow: / User-agent: * Allow: / Sitemap: https://www.pathwren.workers.dev/sitemap.xml