---
title: "Config snippets — AI Crawler Index"
description: "Generated nginx, Caddy, Apache, Cloudflare Worker and Python for classifying and blocking AI crawlers, regenerated from the index so they never drift."
canonical: "https://www.pathwren.workers.dev/snippet/index.html"
url: "https://www.pathwren.workers.dev/snippet/index.md"
format: "markdown"
source: "the bytes of /snippet/index.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# Config snippets

> Generated nginx, Caddy, Apache, Cloudflare Worker and Python for classifying and blocking AI crawlers, regenerated from the index so they never drift.

Generated from the same records as everything else, so they cannot drift from the index.

| Snippet | What it does |
| --- | --- |
| [nginx: classify and block by user-agent](https://www.pathwren.workers.dev/snippet/nginx-map.html) | A map block that turns the user-agent into a category, so you can rate-limit training crawlers, allow AI search, and 403 the disputed ones without maintaining three separate if-chains. |
| [Caddy: block by user-agent](https://www.pathwren.workers.dev/snippet/caddy.html) | A Caddy matcher plus respond directive. Caddy's header_regexp matcher is case-sensitive, so the pattern is written with an inline (?i) flag. |
| [Apache: .htaccess block](https://www.pathwren.workers.dev/snippet/apache-htaccess.html) | BrowserMatchNoCase plus a Require expression. Works on shared hosting where you cannot edit the main config. |
| [Cloudflare Worker: classify at the edge](https://www.pathwren.workers.dev/snippet/cloudflare-worker.html) | Fetches this index's regex list at the edge and tags every request with a category header, so your origin can log AI traffic without shipping a bot list to every app. |
| [Python: classify a request](https://www.pathwren.workers.dev/snippet/python-classify.html) | Forty lines that load agents.json and return the category, operator and robots token for a user-agent string. The shape most log pipelines need. |

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/snippet/index.html)
- [JSON](https://www.pathwren.workers.dev/snippet/index.json)
- [Markdown](https://www.pathwren.workers.dev/snippet/index.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/snippet/index.html](https://www.pathwren.workers.dev/snippet/index.html), generated from that page's own bytes in the same build. The HTML page is canonical.