curl -s https://www.pathwren.workers.dev/category/ai-training.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| anthropic-ai | anthropic-ai | Anthropic | None. Nothing crawls under this name today; keeping the rule is harmless insurance.… |
| Applebot-Extended | Applebot-Extended | Apple | Excluded from Apple Intelligence training. Siri, Spotlight and Safari suggestions are unaf… |
| Bytespider | Bytespider | ByteDance | Little to lose. If you want it gone, expect to block by user-agent at the edge rather than… |
| ClaudeBot | ClaudeBot | Anthropic | Content excluded from training data for future Claude models. No effect on Claude's abilit… |
| cohere-training-data-crawler | cohere-training-data-crawler | Cohere | Excluded from Cohere model training.… |
| Cotoyogi | Cotoyogi | ROIS-DS | Your Japanese-language content is left out of an academic training corpus.… |
| FacebookBot | FacebookBot | Meta | Negligible today. Keep the rule; expect little traffic.… |
| Factset_spyderbot | Factset_spyderbot | FactSet | Exclusion from a financial-data vendor's corpus. Relevant mostly to companies whose filing… |
| Google-Extended | Google-Extended | You are excluded from Gemini grounding and Gemini training. Google Search ranking and inde… | |
| GoogleOther | GoogleOther | No effect on Search indexing. Blocks internal Google research and product fetches.… | |
| GoogleOther-Image | GoogleOther-Image | Google teams outside Search stop fetching your images. Image Search itself is unaffected —… | |
| GoogleOther-Video | GoogleOther-Video | No effect on Search or on Google Video search. Blocks internal Google research fetches of … | |
| GPTBot | GPTBot | OpenAI | Your content is excluded from training data for future OpenAI models. No effect on ChatGPT… |
| ICC-Crawler | ICC-Crawler | NICT | You are excluded from a national research corpus and from the commercial redistributions o… |
| ISSCyberRiskCrawler | ISSCyberRiskCrawler | ISS Corporate Solutions | A rule here is a statement of intent. Your organisation's public footprint still gets scor… |
| Linguee Bot | Linguee Bot | Linguee | Multilingual pages stop feeding a translation corpus. If your site is translated, being in… |
| meta-externalagent | meta-externalagent | Meta | Excluded from Meta AI training. Link previews on Facebook, Instagram and WhatsApp are unaf… |
| Poseidon Research Crawler | Poseidon Research Crawler | Poseidon Research | Exclusion from an interpretability research corpus. No published compliance statement.… |
| QuillBot | QuillBot | QuillBot | Exclusion from QuillBot's corpus. No compliance statement is published, so the rule is a r… |
| Reflectionbot | Reflectionbot | Reflection AI | Unknown by construction — which is itself the reason some people block it. Nothing user-fa… |
| SBIntuitionsBot | SBIntuitionsBot | SB Intuitions | Your content is excluded from a Japanese-language foundation-model corpus. Nothing user-fa… |
| SemrushBot-OCOB | SemrushBot-OCOB | Semrush | Exclusion from Semrush's AI corpus, with its SEO crawl unaffected.… |
| Sidetrade indexer bot | Sidetrade indexer bot | Sidetrade | Exclusion from a commercial B2B dataset. The operator publishes no robots.txt statement, s… |
| TikTokSpider | TikTokSpider | ByteDance | Little to lose unless TikTok search referral matters to you.… |
| Webzio-Extended | Webzio-Extended | Webz.io | Your content is excluded from the AI-training tier of Webz.io's product while ordinary col… |
| YandexAdditional | YandexAdditional | Yandex | You disappear from Yandex's AI answers while staying in Yandex Search. This is Yandex's eq… |
| YandexAdditionalBot | YandexAdditionalBot | Yandex | Same as YandexAdditional: out of Yandex's AI answers, still in Yandex Search. Name both to… |