# Eight ready-made robots.txt files

> Each file names every relevant crawler explicitly, so a later change is a one-line diff and the intent stays readable. Pick the trade-off, curl the file, append it to your robots.txt.

```
curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt >> robots.txt
```

| Policy | Crawlers named | File |
|---|---|---|
| [Allow everything, explicitly](/policy/allow-all.html) | 150 | [/robots/allow-all.txt](/robots/allow-all.txt) |
| [Block AI training, keep AI search](/policy/block-ai-training.html) | 27 | [/robots/block-ai-training.txt](/robots/block-ai-training.txt) |
| [Block every AI crawler](/policy/block-all-ai.html) | 77 | [/robots/block-all-ai.txt](/robots/block-all-ai.txt) |
| [Block corpus and dataset builders](/policy/block-datasets.html) | 17 | [/robots/block-datasets.txt](/robots/block-datasets.txt) |
| [Allow AI search and user fetches, block the rest](/policy/allow-ai-search-only.html) | 65 | [/robots/allow-ai-search-only.txt](/robots/allow-ai-search-only.txt) |
| [Block SEO and backlink crawlers](/policy/block-seo-tools.html) | 17 | [/robots/block-seo-tools.txt](/robots/block-seo-tools.txt) |
| [Block the crawlers with disputed robots compliance](/policy/block-disputed.html) | 18 | [/robots/block-disputed.txt](/robots/block-disputed.txt) |
| [Maximum AI visibility](/policy/maximum-ai-visibility.html) | 133 | [/robots/maximum-ai-visibility.txt](/robots/maximum-ai-visibility.txt) |

---

Machine copies of this listing: [/policy/index.json](/policy/index.json) ·
[/policy/index.html](/policy/index.html) · [everything here](/documents.json) ·
[what changed since your cursor](/changes.json).
`https://www.pathwren.workers.dev/policy` serves this file to a client that ranks `text/markdown` above
`text/html`, the JSON to one that asks for `application/json`, and the page to
everybody else; the canonical is /policy/index.html.

Rebuilt 2026-09-03. An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any of the crawler operators it documents, nor by any other company. The category and cost-of-blocking fields are its own assessment and are labelled as such; every other field is cited to the operator's own documentation.