---
title: "ai-crawler-robots — Does your robots.txt block the crawlers you think it blocks?"
description: "Lint a robots.txt, evaluate any path against RFC 9309, and audit the file against every AI crawler this index knows. Zero dependencies, no network, no URL fetching."
canonical: "https://www.pathwren.workers.dev/ai-crawler-robots/index.html"
url: "https://www.pathwren.workers.dev/ai-crawler-robots/index.md"
format: "markdown"
source: "the bytes of /ai-crawler-robots/index.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T07:55:35+00:00"
license: "CC0-1.0"
---

# ai-crawler-robots

> Lint a robots.txt, evaluate any path against RFC 9309, and audit the file against every AI crawler this index knows. Zero dependencies, no network, no URL fetching.

A robots.txt that looks right and blocks nothing is the normal failure. This reads the file you already have and says which of the 56 AI crawlers it actually stops.

```bash
pip install ai-crawler-robots
```

Lint a robots.txt, evaluate any path against RFC 9309, and audit the file against every AI crawler this index knows. Zero dependencies, no network, no URL fetching.
Python >=3.8,
zero dependencies, standard library only.
Code MIT, bundled data CC0-1.0.

## Command line

| Command | What it does |
| --- | --- |
| `ai-crawler-robots lint robots.txt` | the faults that make a file do nothing: a UA string where a product token belongs, a misspelled directive, Noindex:, a shadowed group, a path without a leading slash |
| `ai-crawler-robots audit robots.txt` | every AI crawler this index knows, split into blocked / allowed / says-no-but-will-not-obey |
| `ai-crawler-robots check robots.txt --agent GPTBot --path /blog/` | allowed or not, and the line number that decided it |
| `ai-crawler-robots diff old.txt new.txt` | what changed by EFFECT, not by text — a reformat is not a change |
| `ai-crawler-robots policy block-ai-training --merge robots.txt` | add a ready-made stance without touching the rules already in the file |

## API

| Call | Returns |
| --- | --- |
| `lint(text)` | findings with level, line, message and a suggested fix |
| `is_allowed(text, agent, path)` | the RFC 9309 verdict, the winning rule, its line, and whether two conformant parsers would disagree |
| `audit(text, path='/')` | blocked / allowed / unenforceable per crawler, plus the tokens in the file that match nothing |
| `diff(old, new)` | only the crawlers whose verdict flipped |
| `merge_policy(text, stance)` | the file with a stance added and its existing groups untouched |
| `refresh()` | today's table, cached six hours — the only network call in the package |

The single most common robots.txt failure is not a syntax error. It is a file that looks right and blocks nothing: a User-agent line carrying a whole browser-style UA string instead of the product token, a group shadowed by an earlier one, a Disallow under a token no crawler sends, or Noindex:, which Google stopped honouring in robots.txt in 2019. Every one of those is invisible to a person reading the file and obvious to a parser.

It reads the file rather than writing one. Generators are everywhere; the question an operator actually has is about the file they already deployed.

Nothing here fetches a URL. There is no argument that takes one, and the only network call in the package is refresh(), which you have to name.

## The data it ships

The wheel bundles [data.json](https://www.pathwren.workers.dev/ai-crawler-robots/data.json) — 150 crawlers,
8 ready-made robots.txt stances, generated 2026-09-03 — so nothing
here needs the network. `refresh()` is the only call that does, it fetches
[the same document](https://www.pathwren.workers.dev/ai-crawler-robots/data.json) live, and it caches for six hours
because that is how often the index behind it is rebuilt.

## Where it comes from

Generated from the [AI Crawler Index](https://www.pathwren.workers.dev/): every crawler record is
checked against its operator's own published documentation, linked on
[each crawler's page](https://www.pathwren.workers.dev/crawler/). On PyPI:
[https://pypi.org/project/ai-crawler-robots/](https://pypi.org/project/ai-crawler-robots/).

Sibling packages, same table, different question: [ai-crawler-index](https://pypi.org/project/ai-crawler-index/) · [ai-crawler-verify](https://pypi.org/project/ai-crawler-verify/) · [ai-crawler-logs](https://pypi.org/project/ai-crawler-logs/).

An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any of the crawler operators it documents, nor by any other company. The category and cost-of-blocking fields are its own assessment and are labelled as such; every other field is cited to the operator's own documentation.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/ai-crawler-robots/index.html)
- [JSON](https://www.pathwren.workers.dev/ai-crawler-robots/index.json)
- [Markdown](https://www.pathwren.workers.dev/ai-crawler-robots/index.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/ai-crawler-robots/index.html](https://www.pathwren.workers.dev/ai-crawler-robots/index.html), generated from that page's own bytes in the same build. The HTML page is canonical.