---
title: "MCP server — AI Crawler Index"
description: "This index as an MCP server: Streamable HTTP at /mcp, nine read-only tools, no key, no signup. Classify a user-agent, look up a crawler, generate robots.txt, check an IP."
canonical: "https://www.pathwren.workers.dev/mcp.html"
url: "https://www.pathwren.workers.dev/mcp.md"
format: "markdown"
source: "the bytes of /mcp.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# MCP server

> This index as an MCP server: Streamable HTTP at /mcp, nine read-only tools, no key, no signup. Classify a user-agent, look up a crawler, generate robots.txt, check an IP.

The whole index, spoken as [Model Context Protocol](https://modelcontextprotocol.io) instead of as files. Streamable HTTP, stateless, no key, no signup, no session
to keep alive. Every tool is read-only.

```text
# it answers right now — no account, no header ceremony
curl -s https://www.pathwren.workers.dev/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | jq -r '.result.tools[].name'
```

## Add it to a client

Anything that speaks MCP over HTTP. Claude Code:

```text
claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp
```

Clients that take a JSON config (Claude Desktop, Cursor, VS Code, Windsurf):

```json
{
 "mcpServers": {
  "ai-crawler-index": { "type": "streamable-http", "url": "https://www.pathwren.workers.dev/mcp" }
 }
}
```

## Tools

| Tool | What it answers |
| --- | --- |
| `classify_user_agent` | Given a raw User-Agent header: which crawler is this, who runs it, what is it for, and what does blocking it cost me? |
| `lookup_crawler` | The full record for one crawler by slug, name or robots.txt token. |
| `list_crawlers` | The 150 crawlers, filtered by category, operator, robots.txt stance or verification method. Returns the valid filter values too, so one call teaches the vocabulary. |
| `generate_robots_txt` | A ready-to-paste robots.txt for one of 8 stances, with the reasoning and the exact tokens it implies. |
| `is_verified_crawler_ip` | Is this IP inside a range the operator itself publishes? 1984 IPv4 and 1062 IPv6 prefixes, mirrored from 15 endpoints, refreshed every six hours. |
| `whats_changed` | Build time, per-source freshness, which sources changed or are failing, and the dated changelog. The cron check for anything that regenerates a config from this data. |

## Resources

Four, each pointing at the file the tools are computed from:
`agents.json`, `ua-regex.json`, `ip-ranges/all.json`,
`status.json`. Read them directly if you would rather do your own matching.

## What it will not do

It has no side effects and no write path: it cannot edit your robots.txt, block anything,
or fetch a URL for you. A user-agent match is a claim, not a proof — `classify_user_agent`
says so on every answer, and `is_verified_crawler_ip` is the tool that turns a claim
into evidence. Where an operator publishes no ranges, no tool here can verify it, and each
one says which case you are in.

## Five more servers on this host

Different jobs, no shared tool names — run all six or just the one you need.
[crawler-log-triage](https://www.pathwren.workers.dev/mcp-triage.html) at `/mcp/triage` takes a whole access
log and returns per-line verdicts, the impersonators, and a robots.txt or WAF ruleset for exactly
what was in it. [agent-discovery-doctor](https://www.pathwren.workers.dev/mcp-doctor.html) at `/mcp/doctor`
turns the question around: which of the 22 discovery documents agents ask for — llms.txt, A2A
agent card, owners.json, mcp.json — does a host actually serve, and who asks for each missing
one. [robots-policy-lint](https://www.pathwren.workers.dev/mcp-robots.html) at `/mcp/robots` reads a
robots.txt you paste and reports what it really does, by RFC 9309 rules.
[crawler-ip-verifier](https://www.pathwren.workers.dev/mcp-netcheck.html) at `/mcp/netcheck` is the batch
and prefix form of `is_verified_crawler_ip`: 500 addresses at a time, CIDR overlap
arithmetic, and a firewall allowlist you can paste.
[mcp-endpoint-lint](https://www.pathwren.workers.dev/mcp-lint.html) at `/mcp/lint` is the one aimed at MCP
itself: give it another server's endpoint and it reports what the handshake negotiated, whether
every advertised tool schema compiles, whether the error codes are the ones JSON-RPC requires,
and what a client finds before it dials — scored out of 100.

Protocol versions 2025-06-18, negotiated per call.
`server/discover` answers for clients on 2026-07-28, `initialize` for everyone else.
Same data as [/data/agents.json](https://www.pathwren.workers.dev/data/agents.json) and
[/openapi.json](https://www.pathwren.workers.dev/openapi.json) — pick whichever your client speaks.
Listed in the [official MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=ai-crawler-index) as `dev.workers.pathwren.www/ai-crawler-index`.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/mcp.html)
- [JSON](https://www.pathwren.workers.dev/mcp.json)
- [Markdown](https://www.pathwren.workers.dev/mcp.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/mcp.html](https://www.pathwren.workers.dev/mcp.html), generated from that page's own bytes in the same build. The HTML page is canonical.