---
title: "MCP server — Crawler IP Verifier"
description: "Check up to 500 addresses at once against operator-published crawler IP ranges, do prefix arithmetic on a CIDR, and export a paste-ready allowlist for nginx, Apache, HAProxy, Cloudflare, ipset or Caddy. Streamable HTTP at /mcp/netcheck, no key."
canonical: "https://www.pathwren.workers.dev/mcp-netcheck.html"
url: "https://www.pathwren.workers.dev/mcp-netcheck.md"
format: "markdown"
source: "the bytes of /mcp-netcheck.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T07:55:35+00:00"
license: "CC0-1.0"
---

# Crawler IP Verifier — MCP server

> Check up to 500 addresses at once against operator-published crawler IP ranges, do prefix arithmetic on a CIDR, and export a paste-ready allowlist for nginx, Apache, HAProxy, Cloudflare, ipset or Caddy. Streamable HTTP at /mcp/netcheck, no key.

A user-agent is a claim anyone can type. The operators who take that seriously publish the
networks their crawlers come from; this host mirrors every one of those lists it can find, and
this server is the batch and prefix arithmetic over the mirror — not one address and a
yes/no, which [/mcp](https://www.pathwren.workers.dev/mcp.html) already answers, but the shapes the question
actually takes in operations.

```text
# a whole log's worth of addresses, in one call
curl -s https://www.pathwren.workers.dev/mcp/netcheck \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_batch",
       "arguments":{"addresses":["20.171.206.1","203.0.113.9"]}}}' \
  | jq -r '.result.structuredContent.results[] | "\(.ip)\t\(.verdict)"'
```

## Tools

| Tool | What it does |
| --- | --- |
| `verify_batch` | Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus a rollup by operator. Accepts {ip, claim} pairs to test a claimed identity against the ranges, and says plainly that a miss is not evidence of a fake. Example: addresses=[{ip:'66.249.66.1', claim:'Googlebot'}, {ip:'203.0.113.9', claim:'GPTBot'}] — the first matches a published Google prefix, the second matches nothing. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot&s=client-dossiers |
| `lookup_prefix` | Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for auditing an allowlist somebody handed you. Give an operator or source slug instead and get everything that source publishes. Example: cidr='66.249.66.0/24'. |
| `export_ip_acl` | Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR list. Every export carries a provenance header naming each source URL and the mirror time, and reports the rule cost. Example: operators='all', format='cidr-list', action='allow'. |
| `verification_methods` | Per crawler: whether the operator publishes IP ranges, documents reverse DNS, or publishes no verification method at all — with the exact source URL or the exact dig commands, and how fresh this mirror is. Call it before reading anything into an address that matched nothing. Reverse DNS is never run here; the command is handed back instead. Example: crawler='claudebot'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verification-methods?crawler=claudebot&s=client-dossiers |
| `range_stats` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Per source: prefix counts, total addresses and /24-equivalents, smallest and largest prefix, the publishing URL and when it was last fetched; plus prefixes published by more than one operator. This is the sizing and freshness information an allowlist decision needs. Example: {} — no arguments at all. |
| `whoami` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what this host's robots policy says about you. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before. |
| `example` | Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — one of its real tools, on a canned input taken from this host's own published data — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. No URL of yours is fetched and no third party is touched. Example: arguments={} runs it and returns the real answer. |

## What is behind it

15 operator-published range endpoints, re-fetched every six hours, unioned
into [/ip-ranges/all.json](https://www.pathwren.workers.dev/ip-ranges/all.json). Every answer carries the source URL
the prefix came from and the minute that source was last fetched, because a range list without
a date is a guess with better formatting. An upstream that is down keeps its last known
prefixes and is marked failed on [/status.json](https://www.pathwren.workers.dev/status.json) rather than silently
shrinking.

## Allowlists you can paste

`export_ip_acl` emits nginx `geo`, nginx allow/deny, Apache
`Require ip`, an HAProxy list, a Cloudflare firewall expression, an
`ipset` script, a Caddy `remote_ip` matcher, or a plain CIDR list —
with a header naming every source and its fetch time, and the rule cost stated up front,
because whether an allowlist is 40 lines or 2,900 decides which of those formats you can
afford.

## What it refuses to claim

**Absence from a range list is not evidence of a fake.** Most crawlers in this
index publish no ranges at all, and several document reverse DNS instead — so a miss means
"cannot be checked this way", not "impostor". `verification_methods` exists to tell
you which of the three cases you are in before you act on a result, and every answer repeats the
distinction rather than assuming you read it once.

**Reverse DNS is never performed here.** This server makes no outbound request
of any kind — no tool takes a URL or a hostname to resolve. Where an operator's documented
method is the reverse/forward lookup, you get the exact `dig` commands and an
explicit "not run", the same refusal [crawler-log-triage](https://www.pathwren.workers.dev/mcp-triage.html)
makes.

## How is this different from /mcp?

`is_verified_crawler_ip` over on [ai-crawler-index](https://www.pathwren.workers.dev/mcp.html) takes
one address and returns a boolean. That is the right tool for one question asked once. This
server takes 500 addresses and returns a table plus a rollup; takes a CIDR and returns
containment and overlap; takes a set of operators and returns a config file. No tool name is
shared with any of the other four servers here.

Protocol versions 2025-06-18, negotiated per call.
`server/discover` answers for clients on 2026-07-28, `initialize` for everyone else.
Read-only, stateless, no key. Listed in the
[official MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=crawler-ip-verifier) as `dev.workers.pathwren.www/crawler-ip-verifier`.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [JSON](https://www.pathwren.workers.dev/mcp-netcheck.json)
- [Markdown](https://www.pathwren.workers.dev/mcp-netcheck.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/mcp-netcheck.html](https://www.pathwren.workers.dev/mcp-netcheck.html), generated from that page's own bytes in the same build. The HTML page is canonical.