---
title: "Python: classify a request — AI Crawler Index"
description: "Forty lines that load agents.json and return the category, operator and robots token for a user-agent string. The shape most log pipelines need."
canonical: "https://www.pathwren.workers.dev/snippet/python-classify.html"
url: "https://www.pathwren.workers.dev/snippet/python-classify.md"
format: "markdown"
source: "the bytes of /snippet/python-classify.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T07:55:35+00:00"
license: "CC0-1.0"
---

# Python: classify a request

> Forty lines that load agents.json and return the category, operator and robots token for a user-agent string. The shape most log pipelines need.

Forty lines that load agents.json and return the category, operator and robots token for a user-agent string. The shape most log pipelines need.

```bash
curl -s https://www.pathwren.workers.dev/snippet/python-classify.txt -o classify.py
```

```text
#!/usr/bin/env python3
"""AI Crawler Index — classify a user-agent string.
curl -s https://www.pathwren.workers.dev/snippet/python-classify.txt -o classify.py

Loads the index once, answers in O(number of crawlers) per call, no dependencies.
Refresh agents.json on whatever schedule you like; the shape never changes.
"""
import json, re, urllib.request

INDEX = "https://www.pathwren.workers.dev/data/agents.json"

def load(url=INDEX):
    with urllib.request.urlopen(url, timeout=20) as r:
        return json.load(r)["crawlers"]

class Classifier:
    def __init__(self, crawlers=None):
        self.crawlers = crawlers or load()
        self.rx = [(re.compile(re.escape(c["user_agent_substring"]), re.I), c)
                   for c in self.crawlers
                   if not c["user_agent_substring"].startswith("(")]

    def __call__(self, ua: str):
        """Return the matching record, or None for anything unrecognised."""
        for rx, c in self.rx:
            if rx.search(ua or ""):
                return c
        return None

if __name__ == "__main__":
    import sys
    c = Classifier()
    for line in (sys.argv[1:] or sys.stdin):
        hit = c(line.strip())
        print(f"{hit['name']}\t{hit['category']}\t{hit['operator']}" if hit
              else "-\tunknown\t-")
```

[raw](https://www.pathwren.workers.dev/snippet/python-classify.txt) · [json](https://www.pathwren.workers.dev/snippet/python-classify.json)

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/snippet/python-classify.html)
- [JSON](https://www.pathwren.workers.dev/snippet/python-classify.json)
- [Markdown](https://www.pathwren.workers.dev/snippet/python-classify.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/snippet/python-classify.html](https://www.pathwren.workers.dev/snippet/python-classify.html), generated from that page's own bytes in the same build. The HTML page is canonical.