---
title: "robots.txt: Block AI training, keep AI search — AI Crawler Index"
description: "Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers."
canonical: "https://www.pathwren.workers.dev/policy/block-ai-training.html"
url: "https://www.pathwren.workers.dev/policy/block-ai-training.md"
format: "markdown"
source: "the bytes of /policy/block-ai-training.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# Block AI training, keep AI search

> Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers.

Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers.

```bash
curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt >> robots.txt
```

The distinction most people actually want, and the one that is easy to get wrong: GPTBot trains, OAI-SearchBot indexes for citation. Blocking both loses you the traffic and gains you nothing extra. Google and Apple have no separate crawler at all — Google-Extended and Applebot-Extended are pure control tokens, so they belong in this file while Googlebot and Applebot must not.

## Names 27 crawlers

[anthropic-ai](https://www.pathwren.workers.dev/crawler/anthropic-ai.html) · [Applebot-Extended](https://www.pathwren.workers.dev/crawler/applebot-extended.html) · [Bytespider](https://www.pathwren.workers.dev/crawler/bytespider.html) · [ClaudeBot](https://www.pathwren.workers.dev/crawler/claudebot.html) · [cohere-training-data-crawler](https://www.pathwren.workers.dev/crawler/cohere-training-data-crawler.html) · [Cotoyogi](https://www.pathwren.workers.dev/crawler/cotoyogi.html) · [FacebookBot](https://www.pathwren.workers.dev/crawler/facebookbot.html) · [Factset_spyderbot](https://www.pathwren.workers.dev/crawler/factset-spyderbot.html) · [Google-Extended](https://www.pathwren.workers.dev/crawler/google-extended.html) · [GoogleOther](https://www.pathwren.workers.dev/crawler/googleother.html) · [GoogleOther-Image](https://www.pathwren.workers.dev/crawler/googleother-image.html) · [GoogleOther-Video](https://www.pathwren.workers.dev/crawler/googleother-video.html) · [GPTBot](https://www.pathwren.workers.dev/crawler/gptbot.html) · [ICC-Crawler](https://www.pathwren.workers.dev/crawler/icc-crawler.html) · [ISSCyberRiskCrawler](https://www.pathwren.workers.dev/crawler/isscyberriskcrawler.html) · [Linguee Bot](https://www.pathwren.workers.dev/crawler/linguee-bot.html) · [meta-externalagent](https://www.pathwren.workers.dev/crawler/meta-externalagent.html) · [Poseidon Research Crawler](https://www.pathwren.workers.dev/crawler/poseidon-research-crawler.html) · [QuillBot](https://www.pathwren.workers.dev/crawler/quillbot.html) · [Reflectionbot](https://www.pathwren.workers.dev/crawler/reflectionbot.html) · [SBIntuitionsBot](https://www.pathwren.workers.dev/crawler/sbintuitionsbot.html) · [SemrushBot-OCOB](https://www.pathwren.workers.dev/crawler/semrushbot-ocob.html) · [Sidetrade indexer bot](https://www.pathwren.workers.dev/crawler/sidetrade-indexer-bot.html) · [TikTokSpider](https://www.pathwren.workers.dev/crawler/tiktokspider.html) · [Webzio-Extended](https://www.pathwren.workers.dev/crawler/webzio-extended.html) · [YandexAdditional](https://www.pathwren.workers.dev/crawler/yandexadditional.html) · [YandexAdditionalBot](https://www.pathwren.workers.dev/crawler/yandexadditionalbot.html)

## The file

[/robots/block-ai-training.txt](https://www.pathwren.workers.dev/robots/block-ai-training.txt) · [json](https://www.pathwren.workers.dev/policy/block-ai-training.json)

```text
# AI Crawler Index — policy: block-ai-training
# Block AI training, keep AI search
# Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers.
# Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/block-ai-training.html
# 27 crawlers named. Paste into robots.txt at your document root.

User-agent: anthropic-ai   # control token, no crawler uses this user-agent
Disallow: /

User-agent: Applebot-Extended   # control token, no crawler uses this user-agent
Disallow: /

User-agent: Bytespider   # compliance disputed; enforce at the edge
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Cotoyogi
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Factset_spyderbot
Disallow: /

User-agent: Google-Extended   # control token, no crawler uses this user-agent
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: GoogleOther-Image
Disallow: /

User-agent: GoogleOther-Video
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: ISSCyberRiskCrawler   # compliance disputed; enforce at the edge
Disallow: /

User-agent: Linguee Bot   # compliance disputed; enforce at the edge
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Poseidon Research Crawler
Disallow: /

User-agent: QuillBot
Disallow: /

User-agent: Reflectionbot
Disallow: /

User-agent: SBIntuitionsBot
Disallow: /

User-agent: SemrushBot-OCOB
Disallow: /

User-agent: Sidetrade indexer bot
Disallow: /

User-agent: TikTokSpider   # compliance disputed; enforce at the edge
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: YandexAdditional
Disallow: /

User-agent: YandexAdditionalBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.pathwren.workers.dev/sitemap.xml
```

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/policy/block-ai-training.html)
- [JSON](https://www.pathwren.workers.dev/policy/block-ai-training.json)
- [Markdown](https://www.pathwren.workers.dev/policy/block-ai-training.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/policy/block-ai-training.html](https://www.pathwren.workers.dev/policy/block-ai-training.html), generated from that page's own bytes in the same build. The HTML page is canonical.