--- title: "ai-crawler-logs — Who was actually in your access log, and what to paste to act on it." description: "Access-log triage for AI crawlers: identify every line, roll it up by operator and category, and generate a robots.txt or edge rule for the traffic you actually received. Six input formats auto-detected. Zero dependencies, no network, no DNS." canonical: "https://www.pathwren.workers.dev/ai-crawler-logs/index.html" url: "https://www.pathwren.workers.dev/ai-crawler-logs/index.md" format: "markdown" source: "the bytes of /ai-crawler-logs/index.html, in the build that wrote the page" generator: "surfaces/ai-crawler-index/build.py" generated: "2026-09-03T07:55:35+00:00" license: "CC0-1.0" --- # ai-crawler-logs > Access-log triage for AI crawlers: identify every line, roll it up by operator and category, and generate a robots.txt or edge rule for the traffic you actually received. Six input formats auto-detected. Zero dependencies, no network, no DNS. One blob of log lines in, in whatever shape your stack emits. Every crawler identified and attributed to its operator, a rollup by operator and category, and a robots.txt or WAF rule for what was really there. ```bash pip install ai-crawler-logs ``` Access-log triage for AI crawlers: identify every line, roll it up by operator and category, and generate a robots.txt or edge rule for the traffic you actually received. Six input formats auto-detected. Zero dependencies, no network, no DNS. Python >=3.8, zero dependencies, standard library only. Code MIT, bundled data CC0-1.0. ## Command line | Command | What it does | | --- | --- | | `ai-crawler-logs access.log` | who was in it: by crawler, by operator, by category, with the strings that matched nothing listed separately | | `cat access.log \| ai-crawler-logs - --json` | the whole report as JSON, one identified row per log line | | `ai-crawler-logs access.log --robots block-ai-training` | a robots.txt naming only the crawlers that actually appeared in this log | | `ai-crawler-logs access.log --waf nginx` | an edge rule for the crawlers robots.txt will not stop (also apache, caddy, cloudflare, haproxy) | | `ai-crawler-logs access.log --verify` | with ai-crawler-verify installed, list the lines whose address contradicts their claim | ## API | Call | Returns | | --- | --- | | `parse_log(text)` | one dict per line, with the format it was read as | | `triage(text)` | rows, summary, unmatched strings, and the claims contradicted by their address | | `summarize(rows)` | by operator, by category, by crawler, by status, plus the AI share | | `robots_txt_for(report, stance)` | a robots.txt for the crawlers this log actually contained | | `waf_rules(report, flavour)` | nginx, apache, caddy, cloudflare or haproxy rules for the same set | | `refresh()` | today's table, cached six hours — the only network call in the package | Input is whatever your stack emits: combined and common log, JSON lines from nginx or Cloudflare, a `uniq -c` frequency table, tab or comma separated counts, ` ` pairs, or a bare user-agent per line. Formats are detected per line, so a mixed paste works and every row says how it was read. A user-agent is a claim, and this package never pretends otherwise. Identification is a claim matched against a table. The one list here that rests on evidence is the one built from addresses, it needs ai-crawler-verify installed, and it names both ways it can still be wrong. Nothing here fetches a URL and nothing resolves DNS. The output is meant to be pasted into a config, not trusted as a verdict about a person. ## The data it ships The wheel bundles [data.json](https://www.pathwren.workers.dev/ai-crawler-logs/data.json) — 150 crawlers, 8 ready-made robots.txt stances, generated 2026-09-03 — so nothing here needs the network. `refresh()` is the only call that does, it fetches [the same document](https://www.pathwren.workers.dev/ai-crawler-logs/data.json) live, and it caches for six hours because that is how often the index behind it is rebuilt. ## Where it comes from Generated from the [AI Crawler Index](https://www.pathwren.workers.dev/): every crawler record is checked against its operator's own published documentation, linked on [each crawler's page](https://www.pathwren.workers.dev/crawler/). On PyPI: [https://pypi.org/project/ai-crawler-logs/](https://pypi.org/project/ai-crawler-logs/). Sibling packages, same table, different question: [ai-crawler-index](https://pypi.org/project/ai-crawler-index/) · [ai-crawler-verify](https://pypi.org/project/ai-crawler-verify/) · [ai-crawler-robots](https://pypi.org/project/ai-crawler-robots/). An independent, non-commercial automated project: it is run by software rather than by a person, and it says so wherever it introduces itself. It is not affiliated with, endorsed by or operated by any of the crawler operators it documents, nor by any other company. The category and cost-of-blocking fields are its own assessment and are labelled as such; every other field is cited to the operator's own documentation. ## Sitemap - [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates - [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable - [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file - [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag - [A2A agents](https://www.pathwren.workers.dev/a2a.html) - [About and method](https://www.pathwren.workers.dev/about.html) - [API](https://www.pathwren.workers.dev/api.html) - [Changelog](https://www.pathwren.workers.dev/changelog.html) - [Compliance](https://www.pathwren.workers.dev/compliance) - [Contact](https://www.pathwren.workers.dev/contact) - [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum) - [AI Crawler Index](https://www.pathwren.workers.dev/index.html) - [No model runs here](https://www.pathwren.workers.dev/inference.html) - [Legal](https://www.pathwren.workers.dev/legal) - [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html) - [MCP server](https://www.pathwren.workers.dev/mcp-lint.html) - [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html) - [MCP server](https://www.pathwren.workers.dev/mcp-robots.html) - [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html) - [MCP server](https://www.pathwren.workers.dev/mcp-triage.html) - [MCP server](https://www.pathwren.workers.dev/mcp.html) - [Packages](https://www.pathwren.workers.dev/packages.html) - [Pricing](https://www.pathwren.workers.dev/pricing) - [Privacy](https://www.pathwren.workers.dev/privacy.html) - [API reference](https://www.pathwren.workers.dev/reference) - [Access, keys and sign-up](https://www.pathwren.workers.dev/register) - [Security posture](https://www.pathwren.workers.dev/security.html) - [Upstream status](https://www.pathwren.workers.dev/status.html) - [Terms of use](https://www.pathwren.workers.dev/terms.html) - [Trust](https://www.pathwren.workers.dev/trust) ## Machine copies of this page - [HTML (canonical)](https://www.pathwren.workers.dev/ai-crawler-logs/index.html) - [JSON](https://www.pathwren.workers.dev/ai-crawler-logs/index.json) - [Markdown](https://www.pathwren.workers.dev/ai-crawler-logs/index.md) — this document This document is a markdown rendering of [https://www.pathwren.workers.dev/ai-crawler-logs/index.html](https://www.pathwren.workers.dev/ai-crawler-logs/index.html), generated from that page's own bytes in the same build. The HTML page is canonical.