---
title: "Cloudflare Worker: classify at the edge — AI Crawler Index"
description: "Fetches this index's regex list at the edge and tags every request with a category header, so your origin can log AI traffic without shipping a bot list to every app."
canonical: "https://www.pathwren.workers.dev/snippet/cloudflare-worker.html"
url: "https://www.pathwren.workers.dev/snippet/cloudflare-worker.md"
format: "markdown"
source: "the bytes of /snippet/cloudflare-worker.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T07:55:35+00:00"
license: "CC0-1.0"
---

# Cloudflare Worker: classify at the edge

> Fetches this index's regex list at the edge and tags every request with a category header, so your origin can log AI traffic without shipping a bot list to every app.

Fetches this index's regex list at the edge and tags every request with a category header, so your origin can log AI traffic without shipping a bot list to every app.

```bash
curl -s https://www.pathwren.workers.dev/snippet/cloudflare-worker.txt -o worker.js
```

```text
// AI Crawler Index — Cloudflare Worker: tag every request with a crawler class.
// curl -s https://www.pathwren.workers.dev/snippet/cloudflare-worker.txt
// Pulls the regex list from this index once per hour and caches it at the edge,
// so the bot list lives in one place instead of in every app.
const INDEX = "https://www.pathwren.workers.dev/data/ua-regex.json";
let cache = null, cachedAt = 0;

async function patterns() {
  if (cache && Date.now() - cachedAt < 3600_000) return cache;
  const r = await fetch(INDEX, { cf: { cacheTtl: 3600 } });
  const d = await r.json();
  cache = Object.entries(d.by_category).map(([k, v]) => [k, new RegExp(v, "i")]);
  cachedAt = Date.now();
  return cache;
}

export default {
  async fetch(request) {
    const ua = request.headers.get("user-agent") || "";
    let klass = "human";
    for (const [name, re] of await patterns()) if (re.test(ua)) { klass = name; break; }
    const req = new Request(request);
    req.headers.set("X-Crawler-Class", klass);
    return fetch(req);
  }
};
```

[raw](https://www.pathwren.workers.dev/snippet/cloudflare-worker.txt) · [json](https://www.pathwren.workers.dev/snippet/cloudflare-worker.json)

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/snippet/cloudflare-worker.html)
- [JSON](https://www.pathwren.workers.dev/snippet/cloudflare-worker.json)
- [Markdown](https://www.pathwren.workers.dev/snippet/cloudflare-worker.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/snippet/cloudflare-worker.html](https://www.pathwren.workers.dev/snippet/cloudflare-worker.html), generated from that page's own bytes in the same build. The HTML page is canonical.