---
title: "MCP server — Crawler Log Triage"
description: "Paste raw access-log lines into an MCP tool: every crawler identified and attributed, the impersonators named against operator-published IP ranges, and robots.txt or WAF rules for exactly what you saw. Streamable HTTP at /mcp/triage, no key, no signup."
canonical: "https://www.pathwren.workers.dev/mcp-triage.html"
url: "https://www.pathwren.workers.dev/mcp-triage.md"
format: "markdown"
source: "the bytes of /mcp-triage.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T07:55:35+00:00"
license: "CC0-1.0"
---

# Crawler Log Triage — MCP server

> Paste raw access-log lines into an MCP tool: every crawler identified and attributed, the impersonators named against operator-published IP ranges, and robots.txt or WAF rules for exactly what you saw. Streamable HTTP at /mcp/triage, no key, no signup.

Paste your access log. Get back what is crawling you, who runs it, what it costs to
block, and which lines are lying about who they are. Streamable HTTP, stateless, no key,
no signup. Every tool is read-only.

```text
# three lines of a log, one call, no account
curl -s https://www.pathwren.workers.dev/mcp/triage \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_log",
       "arguments":{"log":"66.249.66.1 Googlebot/2.1\n203.0.113.9 Googlebot/2.1\n412 GPTBot/1.2"}}}' \
  | jq -r '.result.structuredContent.rows[] | "\(.identified)\t\(.operator)\t\(.verdict)"'

Googlebot	Google	verified
Googlebot	Google	impersonation
GPTBot	OpenAI	unverifiable
```

With a real file, the shape you actually want:

```bash
curl -s https://www.pathwren.workers.dev/mcp/triage \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d "$(jq -Rs '{jsonrpc:"2.0",id:1,method:"tools/call",
       params:{name:"summarize_by_operator",arguments:{log:.}}}' < access.log)" \
  | jq -r '.result.structuredContent.by_operator[] | "\(.requests)\t\(.operator)\t\(.share_pct)%"'
```

## Add it to a client

```text
claude mcp add --transport http crawler-log-triage https://www.pathwren.workers.dev/mcp/triage
```

Clients that take a JSON config (Claude Desktop, Cursor, VS Code, Windsurf):

```json
{
 "mcpServers": {
  "crawler-log-triage": { "type": "streamable-http", "url": "https://www.pathwren.workers.dev/mcp/triage" }
 }
}
```

## Tools

| Tool | What it answers |
| --- | --- |
| `triage_log` | The whole job. Per line: which crawler, whose, what for, what blocking costs — plus a rollup by operator and category and every line whose address contradicts its claim. |
| `find_impersonators` | Only the lines claiming a crawler whose operator publishes ranges, from an address in none of them. Checked against 1984 IPv4 and 1062 IPv6 prefixes from 15 operator sources. |
| `summarize_by_operator` | Aggregate only: who crawled you, how many requests, what share, which category, what blocking each would cost. Eats a `uniq -c` table straight from a shell pipeline. |
| `robots_from_log` | A robots.txt naming only the crawlers your log actually contains — not a generic list of everything that exists — with a warning for any that do not documentably obey it. |
| `waf_ruleset_from_log` | nginx, Caddy, Cloudflare, HAProxy or Apache rules for the same set. The enforcement half, for the crawlers that ignore robots.txt. |

## What it eats

Formats are detected per line, so a messy paste is fine and a mixed file is fine:

| Shape | Example |
| --- | --- |
| combined / common | `66.249.66.1 - - [01/Sep/2026:00:14:02 +0000] "GET / HTTP/1.1" 200 512 "-" "…Googlebot/2.1…"` |
| JSON lines | `{"remote_addr":"66.249.66.1","http_user_agent":"…Googlebot/2.1…"}` |
| `uniq -c` table | `412 Mozilla/5.0 … GPTBot/1.2 …` — the count is respected, so 412 lines cost you one |
| ip + user-agent | `66.249.66.1 Googlebot/2.1` |
| bare user-agent | `Mozilla/5.0 (compatible; ClaudeBot/1.0; …)` |

The usual way to produce the third one:
`awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -50`

## How is this different from [/mcp](https://www.pathwren.workers.dev/mcp.html)?

[ai-crawler-index](https://www.pathwren.workers.dev/mcp.html) answers questions about *one* thing — this
user-agent, this IP, this crawler, this stance. It is a reference. This server does not answer
questions about crawlers at all: it takes *your evidence*, a log you already have, and
gives back a decision about it. One call in, per-line verdicts plus a rollup plus a config out.
Different input, different unit of work, no tool in common — same index underneath, so the two
agree by construction. Run both.

## What it will not do

It has no side effects and no write path. **It has no tool that fetches a URL, and no tool that takes one** — it works on text you paste, and nothing else. That is deliberate:
a log-analysis tool that will fetch an arbitrary URL for whoever is talking to it is an SSRF
proxy with a friendly description, so this one cannot be asked.

Operators who verify by reverse DNS (24 of them, including Bing, Apple and
DuckDuckGo) cannot be checked without a lookup, so instead of doing one it hands you the exact
`dig` command and says the result is unknown until you run it. A user-agent is a
claim, not a proof; every answer says which of the two you are holding.

Protocol versions 2025-06-18, negotiated per call.
`server/discover` answers for clients on 2026-07-28, `initialize` for everyone else.
Same index as [/data/agents.json](https://www.pathwren.workers.dev/data/agents.json), refreshed every six hours.
Listed in the [official MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=crawler-log-triage) as `dev.workers.pathwren.www/crawler-log-triage`.

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/mcp-triage.html)
- [JSON](https://www.pathwren.workers.dev/mcp-triage.json)
- [Markdown](https://www.pathwren.workers.dev/mcp-triage.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/mcp-triage.html](https://www.pathwren.workers.dev/mcp-triage.html), generated from that page's own bytes in the same build. The HTML page is canonical.