# AI Crawler Index > Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 150 crawlers, 74 operators, 1984 IPv4 and 1062 IPv6 prefixes mirrored from 15 endpoints the operators publish themselves. Static files, no key, no rate limit, CORS open, CC0. Rebuilt 2026-09-03. The question this exists to answer: *which of these do I allow, and what do I lose if I do not?* Operators document that a bot exists. They do not document what blocking it costs you. Every record here carries that, in one field, in plain language. ## Start here - [All 150 crawlers](/data/agents.json): the whole dataset, one file. - [Tools you can call with a URL](/tools/index.html): the read-only MCP tools of this host, as keyless GET endpoints. No account, no key, no JSON-RPC. - [Who actually crawls this host](/bot/index.html): 163 named clients — bots, registry probes, trust scanners and liveness checkers — one page each, generated from this host's own request log: the exact user-agent, first and last seen, how many addresses it came from, the paths it asked for in order, the status codes it got, and what it asked for that did not exist. Window 2026-08-31T20:58:11+00:00 to 2026-09-03T06:25:38+00:00 UTC. Most of these names are documented nowhere else on the web. Whole set in one request: [/data/observed-clients.json](/data/observed-clients.json), flat table at [/data/observed-clients.csv](/data/observed-clients.csv). - [Ready-made robots.txt](/policy/index.html): eight policies, each a file you can curl. - [Published IP ranges](/ip-ranges/all.json): every operator prefix list, one schema. - [Upstream status](/status.json): which operator endpoints answered, and when. - [What changed since your last read](/changes.json?since=0): a cursor feed. Read `cursor` from the response, send it back as `since`, and you get only what moved — prefixes added or removed, upstreams that failed or recovered, records edited. The cursor advances only when something really changed, so an unchanged answer is proof rather than luck. If you keep one file from this host up to date, keep it with this instead of re-downloading. - [OpenAPI 3.1](/openapi.json): every read endpoint, described. - [API reference](/reference): the same endpoints as a page, each with the curl that calls it, generated from that OpenAPI document rather than typed. No JavaScript on it. Also served at /docs, /documentation, /api and /api/docs; as data at [/reference.json](/reference.json). - [API version root](/api/v1): which versions of this API exist, and the endpoint list cut from the same OpenAPI document. There is one version, `v1`, and the prefix is an ADDRESS: `/api/v1/data/agents.json` returns the same bytes as `/data/agents.json`, with a canonical link home. `/api/v2` and `/api/v3` are 404 by decision and say so in problem+json — the full history is at [/api/versions](/api/versions). ## Polling this host A cursor is an integer the change feed hands back in every answer it gives you. Send it as `?since=` on the next request and you get only what happened after it: `GET /changes.json?since=0` once, then `?since=` from then on, for ever. The cursor advances only when the published data really differs from what was published before — an upstream that reissues a byte-identical file does not move it — so an answer with nothing in it is proof that nothing changed rather than a request that missed something. That empty answer is about 2.5 KB against the 3046 prefixes and 150 records it saves you re-reading, and if you send back the `ETag` from your previous response as `If-None-Match` — or the `Last-Modified` as `If-Modified-Since`, both are honoured on every document here — it is a 304 with no body at all: staying current costs you strictly less than finding out you already were. The operator endpoints behind it are re-fetched every six hours, so six hours is the interval that always has news; anything from ten minutes up is welcome, nothing here is rate limited and no key is needed, but polling faster than the data moves returns the cursor you already hold. This host itself has no webhook and no socket to offer you — the feed is the whole mechanism here — and every JSON document links it as `"rel": "changes"` with a `Link: <...>; rel="changes"` header beside it, and the same URL again as `rel="alternate"` for a reader that only follows registered relations. The next refresh is due at 2026-09-03T08:42:02+00:00, and you do not have to remember that: every machine document on this host carries `Cache-Control: max-age=` and an `X-Next-Refresh: 2026-09-03T08:42:02+00:00` header, both counting down to the same instant. A poller can sleep on `max-age` and wake exactly when there is something new. If the refresh is late the countdown floors at 300 seconds rather than inventing a date; /status.json says which upstreams answered. The one push that exists here is somebody else's: /feed.json and /feed.xml carry `Link: ; rel="hub"` beside `rel="self"`, and the feed bodies name the same hub — which is what a hub checks a subscription against. This loop pings that hub on its own six-hourly pass, publishing the channel-tagged copy of the feed as the topic; if you subscribe there, delivery is the hub's business and not a poll against us. ## Machine endpoints - [/documents.json](/documents.json): the document ledger — every URL this host publishes, each with a strong ETag and the date its bytes last changed. Read it once and re-fetch only what moved; you never have to HEAD 450 files to find out that none of them did. TSV form at [/documents.txt](/documents.txt) for `awk`. - [/data/agents.json](/data/agents.json): every record, with categories and an endpoint map. - [/data/agents.csv](/data/agents.csv): the same table, flat. - [/data/user-agents.txt](/data/user-agents.txt): user-agent substrings, one per line. - [/data/robots-tokens.txt](/data/robots-tokens.txt): robots.txt tokens, one per line. - [/data/ua-regex.json](/data/ua-regex.json): pre-escaped regex, whole-list and per category. - [/data/ip-sources.json](/data/ip-sources.json): which operators publish ranges, and where. - [/ip-ranges/all.txt](/ip-ranges/all.txt): every published CIDR, one per line, for a WAF. - [/status.json](/status.json): freshness of every upstream source. - [/changes.json](/changes.json?since=0): the since-cursor changes feed — `?since=`, `&limit=` up to 400. Strong ETag and Last-Modified on every answer: send either back and an unchanged feed costs you a 304 with no body. The upstreams are re-fetched every six hours, so polling faster than that returns the same cursor. - [/crawler/.json](/crawler/gptbot.json): one record per crawler. - [/crawler/.md](/crawler/gptbot.md): the same record as markdown, one file each. - [/policy/.json](/policy/block-ai-training.json) and [/robots/.txt](/robots/block-ai-training.txt). - [/feed.json](/feed.json), [/feed.xml](/feed.xml): what changed. - [/mcp](/mcp.html): the same data as an MCP server (Streamable HTTP, no key, nine tools). If you are an agent that speaks MCP, connect to `https://www.pathwren.workers.dev/mcp` and call `tools/list`. - [/mcp/triage](/mcp-triage.html): a second MCP server, `https://www.pathwren.workers.dev/mcp/triage`. It takes a whole access log and returns per-line verdicts, the impersonators, and a robots.txt or WAF ruleset for exactly what was in it. No tool in common with /mcp — different input, different job. - [/mcp/doctor](/mcp-doctor.html): a third MCP server, `https://www.pathwren.workers.dev/mcp/doctor`. It checks which of the 22 discovery documents agents actually ask for — llms.txt, agent card, owners.json, oauth metadata, mcp.json, apis.json — a host serves, names who asks for each missing one, validates a pasted llms.txt or agent card, and drafts an llms.txt from a sitemap. It refuses to check this host or any private one. - [/mcp/robots](/mcp-robots.html): a fourth MCP server, `https://www.pathwren.workers.dev/mcp/robots`. It reads a robots.txt you paste — the only server here that takes one as input rather than writing one — and reports the faults that make a file mean less than its author thought (misspelled directives, `noindex:`, a user-agent string where a product token belongs), evaluates paths by RFC 9309 matching, and says which AI crawlers the file actually blocks. - [/mcp/netcheck](/mcp-netcheck.html): a fifth MCP server, `https://www.pathwren.workers.dev/mcp/netcheck`. Batch address verification against the mirrored operator prefix lists (500 at a time), containment and overlap arithmetic on a CIDR, and paste-ready allowlists for nginx, Apache, HAProxy, Cloudflare, ipset or Caddy. It never resolves anything: reverse DNS is handed back as a command, never a result. - [/mcp/lint](/mcp-lint.html): a sixth MCP server, `https://www.pathwren.workers.dev/mcp/lint`. Point it at any MCP endpoint and it speaks JSON-RPC to it: does `initialize` negotiate the version it was asked for and survive a second connection, does every advertised tool carry a valid JSON Schema, does an unknown method return `-32601` or an HTML 500, and which pre-dial discovery documents answer — including `/.well-known/oauth-protected-resource` in the RFC 9728 §3.1 insertion form most hosts omit. `score_card` does all four in one call, 18 requests, out of 100. Called with no arguments it lints a broken fixture inside the worker and touches nobody; it refuses to fetch this host, private names or IP literals before opening a socket. - [/a2a](/a2a.html): the same nine skills as an A2A (Agent2Agent) v1.0 agent — JSON-RPC 2.0 over POST, no key. Agent card at [/.well-known/agent-card.json](/.well-known/agent-card.json), identical bytes at the legacy [/.well-known/agent.json](/.well-known/agent.json) and from `GET /a2a`. **`message/send` returns a Task already in state `completed`, in the same response** — nothing to poll, nothing to subscribe to. The task id encodes the call, so `tasks/get` re-runs it rather than reading a store this host does not keep; an id we never minted answers `-32001` and hands back the `message/send` body that would create one. No streaming, no push notifications, no task store, and the card declares each of those false. - [/a2a/example.json](/a2a/example.json): **every A2A skill on this host as a complete request you can send unedited** — all eight agents, 48 skills, each with the whole JSON-RPC body and a curl line. If you have nothing to hand, send `whoami`: it takes no arguments and it answers about you — the user-agent you sent, the address you came from, the class this host's instrument books you as, and what our robots policy says about you. ## The tools here, callable without an MCP client Every read-only tool this host runs over MCP also answers a plain `GET`: no account, no key, no handshake, no JSON-RPC. Same implementation behind both doors — the HTTP endpoint calls the MCP tool in-process — so the two cannot answer differently. CORS open, cacheable, CC0. Catalogue: [/tools/index.html](/tools/index.html) ([json](/tools/index.json), [md](/tools/index.md)). - [/tools/whoami](/tools/whoami.html): **Needs nothing from you.** Takes no arguments at all. It answers about the request you just made: the user-agent you sent, the address you came from, whether the crawler index recognises you, whether any operator publishes a prefix containing your address, the class this host's own instrument books you as, and whether we have seen you here before and what you fetched. Every fact comes from your own request headers or from a file this host already publishes; nothing is fetched and nothing about you is invented. `curl -s 'https://www.pathwren.workers.dev/tools/whoami?s=client-dossiers'` - [/tools/example](/tools/example.html): **Needs nothing from you.** Takes no arguments at all. It runs this host's flagship tool (classify_user_agent) on a canned input from our own published log — the busiest client we have seen — and returns exactly the answer a real call returns, not a mock. Use it to see the shape of an answer before deciding what to send. `curl -s 'https://www.pathwren.workers.dev/tools/example?s=client-dossiers'` - [/tools/verify-crawler](/tools/verify-crawler.html): Check one or more IP addresses against every crawler prefix the operators themselves publish (Google, OpenAI, Anthropic, Microsoft, Apple, Perplexity, Amazon and the rest, remirrored every six hours), and — when you name the crawler the client claimed to be — say whether the range that contains it belongs to that operator or to a different one. `curl -s 'https://www.pathwren.workers.dev/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot&s=client-dossiers'` - [/tools/robots-lint](/tools/robots-lint.html): Paste a robots.txt and get every fault that makes it do something other than what it looks like: misspelled directives, a whole User-Agent string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, a byte-order mark. Each finding carries the line number and the fix. `curl -s 'https://www.pathwren.workers.dev/tools/robots-lint?robots_txt=User-agent%3A%20GPTBot%0ADisallow%3A%20%2F%0ANoindex%3A%20%2Fprivate%0A&s=client-dossiers'` - [/tools/robots-allowed](/tools/robots-allowed.html): Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309 — longest token match for the group, longest pattern for the rule, Allow breaking a tie, * and $ supported — and get allowed/disallowed per path with the exact line that decided it, plus a flag on the cases where a merge-groups parser and a first-group-wins parser would disagree. `curl -s 'https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=User-agent%3A%20*%0ADisallow%3A%20%2F%0AAllow%3A%20%2Fblog%2F%0A&ua=GPTBot&path=/blog/post&path=/private&s=client-dossiers'` - [/tools/ai-access](/tools/ai-access.html): Evaluate a pasted robots.txt against every AI crawler in this index and get the two lists that matter — blocked and allowed, per operator and category — plus the tokens in your file that match no known crawler (a typo blocks nothing), and the separation between crawlers that document obedience and the ones observed ignoring robots.txt, which need an IP or WAF rule instead. `curl -s 'https://www.pathwren.workers.dev/tools/ai-access?robots_txt=User-agent%3A%20GPTBot%0ADisallow%3A%20%2F%0A&path=/&s=client-dossiers'` - [/tools/classify-ua](/tools/classify-ua.html): Identify the crawler behind a raw User-Agent header: operator, category, robots.txt token and stance, how it can be verified, and what blocking it costs. An unmatched string is reported as unmatched, with a labelled substring heuristic and never an identification. `curl -s 'https://www.pathwren.workers.dev/tools/classify-ua?ua=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko%3B%20compatible%3B%20GPTBot%2F1.2%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot)&s=client-dossiers'` - [/tools/verification-methods](/tools/verification-methods.html): Before you read anything into a miss from /tools/verify-crawler: which crawlers can be verified by published prefix, which document forward-confirmed reverse DNS instead (with the exact two dig commands to run), and which publish no verification method at all — with the mirror's own counts and fetch dates. `curl -s 'https://www.pathwren.workers.dev/tools/verification-methods?crawler=claudebot&s=client-dossiers'` The same tools over MCP if you do speak it: `tools/call` at https://www.pathwren.workers.dev/mcp — and every MCP server here also publishes `whoami` and `example`, both of which take `arguments: {}`, so you can make a real call before you have decided what to ask. ## Every client observed asking this host for something One page each, generated from the request log. `not observed` means exactly that: we publish what we measured and never a guess, and nothing here is a claim about intent. - [GPTBot](/bot/gptbot.html) ([json](/bot/gptbot.json), [md](/bot/gptbot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai` — 1835 request(s) from 11 address(es), 2026-09-01T05:23:47Z to 2026-09-03T06:00:46Z. - [SentinelOracle](/bot/sentineloracle.html) ([json](/bot/sentineloracle.json), [md](/bot/sentineloracle.md)): `SentinelOracle/0.1 (+https://glimind.com/opt-out; liveness-only, never invokes tools)` — 1463 request(s) from 1 address(es), 2026-09-01T01:24:24Z to 2026-09-03T06:23:38Z. - [ClaudeBot](/bot/claudebot.html) ([json](/bot/claudebot.json), [md](/bot/claudebot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@a` — 1342 request(s) from 6 address(es), 2026-09-01T00:46:32Z to 2026-09-03T06:00:46Z. - [archive.org_bot](/bot/archive-org-bot.html) ([json](/bot/archive-org-bot.json), [md](/bot/archive-org-bot.md)): `Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno` — 993 request(s) from 7 address(es), 2026-08-31T20:58:34Z to 2026-09-03T03:05:14Z. - [node](/bot/node.html) ([json](/bot/node.json), [md](/bot/node.md)): `node` — 929 request(s) from 53 address(es), 2026-08-31T22:49:52Z to 2026-09-03T06:25:34Z. - [mcpbeat](/bot/mcpbeat.html) ([json](/bot/mcpbeat.json), [md](/bot/mcpbeat.md)): `mcpbeat/0.1 (+https://mcpbeat.com/bot/; liveness check)` — 456 request(s) from 1 address(es), 2026-09-01T01:43:38Z to 2026-09-03T06:25:38Z. - [YandexBot](/bot/yandexbot.html) ([json](/bot/yandexbot.json), [md](/bot/yandexbot.md)): `Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)` — 434 request(s) from 252 address(es), 2026-08-31T20:58:12Z to 2026-09-03T02:29:48Z. - [ShapBot](/bot/shapbot.html) ([json](/bot/shapbot.json), [md](/bot/shapbot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0` — 346 request(s) from 7 address(es), 2026-09-02T02:31:06Z to 2026-09-02T02:34:43Z. - [DataForSeoBot](/bot/dataforseobot.html) ([json](/bot/dataforseobot.json), [md](/bot/dataforseobot.md)): `Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)` — 246 request(s) from 1 address(es), 2026-09-03T03:40:24Z to 2026-09-03T06:03:26Z. - [agent-tools.cloud-crawler](/bot/agent-tools-cloud-crawler.html) ([json](/bot/agent-tools-cloud-crawler.json), [md](/bot/agent-tools-cloud-crawler.md)): `agent-tools.cloud-crawler/0.1 (+https://agent-tools.cloud)` — 233 request(s) from 1 address(es), 2026-09-01T03:55:53Z to 2026-09-03T04:21:57Z. - [402explorer](/bot/402explorer.html) ([json](/bot/402explorer.json), [md](/bot/402explorer.md)): `402explorer/0.1 (+https://discover.paygent.net/about)` — 211 request(s) from 1 address(es), 2026-09-01T04:08:29Z to 2026-09-03T06:00:34Z. - [AgentDisco](/bot/agentdisco.html) ([json](/bot/agentdisco.json), [md](/bot/agentdisco.md)): `AgentDisco/1.0 (+https://agentdisco.io/bot)` — 197 request(s) from 1 address(es), 2026-09-01T02:28:30Z to 2026-09-02T06:59:26Z. - [undici](/bot/undici.html) ([json](/bot/undici.json), [md](/bot/undici.md)): `undici` — 160 request(s) from 3 address(es), 2026-09-01T00:36:09Z to 2026-09-03T06:01:15Z. - [AgentGrade](/bot/agentgrade.html) ([json](/bot/agentgrade.json), [md](/bot/agentgrade.md)): `AgentGrade/1.0 (security research; agentgrade.net)` — 155 request(s) from 1 address(es), 2026-09-01T02:53:17Z to 2026-09-03T02:53:44Z. - [agent-ready-scanner](/bot/agent-ready-scanner.html) ([json](/bot/agent-ready-scanner.json), [md](/bot/agent-ready-scanner.md)): `agent-ready-scanner/1.0 (+https://agent-ready.dev)` — 130 request(s) from 1 address(es), 2026-09-03T00:22:52Z to 2026-09-03T00:22:55Z. - [aisec-registry](/bot/aisec-registry.html) ([json](/bot/aisec-registry.json), [md](/bot/aisec-registry.md)): `aisec-registry/0.2 (+https://sec.sqrx.io)` — 100 request(s) from 5 address(es), 2026-09-01T09:17:24Z to 2026-09-02T09:13:55Z. - [Waggle](/bot/waggle.html) ([json](/bot/waggle.json), [md](/bot/waggle.md)): `Waggle/1.0 (+https://waggle.zone)` — 98 request(s) from 39 address(es), 2026-09-02T04:15:47Z to 2026-09-03T05:35:58Z. - [AgentGaugeBot](/bot/agentgaugebot.html) ([json](/bot/agentgaugebot.json), [md](/bot/agentgaugebot.md)): `AgentGaugeBot/1.0 (+https://agentgauge.ai; agent-readiness scanner)` — 87 request(s) from 1 address(es), 2026-09-03T05:27:45Z to 2026-09-03T05:37:54Z. - [AgentTrust-Monitor](/bot/agenttrust-monitor.html) ([json](/bot/agenttrust-monitor.json), [md](/bot/agenttrust-monitor.md)): `AgentTrust-Monitor/1.0 (+https://agenttrust.site/methodology)` — 79 request(s) from 1 address(es), 2026-09-01T03:08:31Z to 2026-09-03T06:16:27Z. - [A2A-Registry-HealthCheck](/bot/a2a-registry-healthcheck.html) ([json](/bot/a2a-registry-healthcheck.json), [md](/bot/a2a-registry-healthcheck.md)): `A2A-Registry-HealthCheck/1.0` — 68 request(s) from 12 address(es), 2026-09-01T02:52:12Z to 2026-09-03T06:02:49Z. - [Enerlio](/bot/enerlio.html) ([json](/bot/enerlio.json), [md](/bot/enerlio.md)): `Enerlio GmbH FACTANKER marc@enerlio.de` — 62 request(s) from 1 address(es), 2026-09-01T01:07:45Z to 2026-09-03T01:06:29Z. - [Mozilla](/bot/mozilla.html) ([json](/bot/mozilla.json), [md](/bot/mozilla.md)): `Mozilla/5.0` — 60 request(s) from 6 address(es), 2026-09-01T02:49:36Z to 2026-09-02T02:05:57Z. - [python-httpx](/bot/python-httpx.html) ([json](/bot/python-httpx.json), [md](/bot/python-httpx.md)): `python-httpx/0.28.1` — 55 request(s) from 13 address(es), 2026-08-31T22:44:55Z to 2026-09-03T06:12:53Z. - [Deno](/bot/deno.html) ([json](/bot/deno.json), [md](/bot/deno.md)): `Deno/2.7.5` — 45 request(s) from 12 address(es), 2026-08-31T22:30:06Z to 2026-09-02T09:32:00Z. - [APIEvangelist](/bot/apievangelist.html) ([json](/bot/apievangelist.json), [md](/bot/apievangelist.md)): `Mozilla/5.0 (compatible; APIEvangelist/1.0)` — 44 request(s) from 1 address(es), 2026-09-01T11:56:42Z to 2026-09-01T12:06:54Z. - [AgenstryBot](/bot/agenstrybot.html) ([json](/bot/agenstrybot.json), [md](/bot/agenstrybot.md)): `AgenstryBot/0.3.0 (+https://agenstry.com/bot)` — 43 request(s) from 1 address(es), 2026-09-01T04:28:51Z to 2026-09-03T02:59:00Z. - [ProofBench](/bot/proofbench.html) ([json](/bot/proofbench.json), [md](/bot/proofbench.md)): `ProofBench/0.1 (+https://proofbench.dev/about/probe; MCP registry health probe)` — 42 request(s) from 1 address(es), 2026-08-31T22:17:01Z to 2026-09-03T04:34:01Z. - [mcpscan](/bot/mcpscan.html) ([json](/bot/mcpscan.json), [md](/bot/mcpscan.md)): `mcpscan/1.0 (+https://modc2.com/mcpscan; MCP index crawler)` — 40 request(s) from 1 address(es), 2026-09-01T02:28:50Z to 2026-09-03T03:20:14Z. - [curl](/bot/curl.html) ([json](/bot/curl.json), [md](/bot/curl.md)): `curl/8.7.1` — 38 request(s) from 4 address(es), 2026-09-01T02:28:30Z to 2026-09-02T11:32:11Z. - [MCPWatch](/bot/mcpwatch.html) ([json](/bot/mcpwatch.json), [md](/bot/mcpwatch.md)): `MCPWatch/0.1.0 (+mcpwatch@iyre.com) longitudinal MCP security research` — 38 request(s) from 1 address(es), 2026-09-01T03:49:49Z to 2026-09-03T03:53:31Z. - [GolemreachTrustBot](/bot/golemreachtrustbot.html) ([json](/bot/golemreachtrustbot.json), [md](/bot/golemreachtrustbot.md)): `GolemreachTrustBot/0.1 (+https://golemreach.com/trust/bot)` — 37 request(s) from 1 address(es), 2026-09-01T00:48:12Z to 2026-09-03T06:22:00Z. - [python-httpx2](/bot/python-httpx2.html) ([json](/bot/python-httpx2.json), [md](/bot/python-httpx2.md)): `python-httpx2/2.12.0` — 36 request(s) from 4 address(es), 2026-08-31T22:15:20Z to 2026-09-02T08:58:41Z. - [python-requests](/bot/python-requests.html) ([json](/bot/python-requests.json), [md](/bot/python-requests.md)): `python-requests/2.33.0` — 33 request(s) from 30 address(es), 2026-09-01T06:23:35Z to 2026-09-02T07:52:09Z. - [SaSame-MCP-Audit](/bot/sasame-mcp-audit.html) ([json](/bot/sasame-mcp-audit.json), [md](/bot/sasame-mcp-audit.md)): `SaSame-MCP-Audit/0.1` — 31 request(s) from 1 address(es), 2026-09-01T01:07:30Z to 2026-09-03T06:01:15Z. - [Go-http-client](/bot/go-http-client.html) ([json](/bot/go-http-client.json), [md](/bot/go-http-client.md)): `Go-http-client/2.0` — 30 request(s) from 2 address(es), 2026-08-31T23:00:06Z to 2026-09-03T06:12:46Z. - [gtm-engine](/bot/gtm-engine.html) ([json](/bot/gtm-engine.json), [md](/bot/gtm-engine.md)): `gtm-engine/0.1 (+contact enrichment; no pattern-guessing)` — 30 request(s) from 1 address(es), 2026-09-01T16:51:55Z to 2026-09-01T16:52:07Z. - [Neuronto](/bot/neuronto.html) ([json](/bot/neuronto.json), [md](/bot/neuronto.md)): `Neuronto/1.0 (+https://neuronto.com/about; ARD registry; crawler)` — 27 request(s) from 2 address(es), 2026-09-01T08:18:38Z to 2026-09-03T05:23:57Z. - [apis.io-submit](/bot/apis-io-submit.html) ([json](/bot/apis-io-submit.json), [md](/bot/apis-io-submit.md)): `apis.io-submit/1.0 (+https://apis.io)` — 25 request(s) from 3 address(es), 2026-08-31T21:21:01Z to 2026-09-02T04:49:24Z. - [NotHumanSearch](/bot/nothumansearch.html) ([json](/bot/nothumansearch.json), [md](/bot/nothumansearch.md)): `NotHumanSearch/1.0 (+https://nothumansearch.ai/about)` — 25 request(s) from 1 address(es), 2026-09-03T04:08:25Z to 2026-09-03T05:06:48Z. - [Python](/bot/python.html) ([json](/bot/python.json), [md](/bot/python.md)): `Python/3.11 aiohttp/3.13.5` — 24 request(s) from 18 address(es), 2026-09-03T01:27:04Z to 2026-09-03T02:34:42Z. - [Amazonbot](/bot/amazonbot.html) ([json](/bot/amazonbot.json), [md](/bot/amazonbot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://dev` — 23 request(s) from 23 address(es), 2026-09-02T04:57:56Z to 2026-09-03T05:57:16Z. - [ColonistOne](/bot/colonistone-2.html) ([json](/bot/colonistone-2.json), [md](/bot/colonistone-2.md)): `ColonistOne/1.0 (autonomous agent; review for thecolony.ai)` — 23 request(s) from 1 address(es), 2026-09-03T05:35:05Z to 2026-09-03T05:38:53Z. - [davefeedread](/bot/davefeedread.html) ([json](/bot/davefeedread.json), [md](/bot/davefeedread.md)): `davefeedread v0.5.25` — 23 request(s) from 1 address(es), 2026-09-03T01:08:27Z to 2026-09-03T06:20:57Z. - [io.verifymcp](/bot/io-verifymcp.html) ([json](/bot/io-verifymcp.json), [md](/bot/io-verifymcp.md)): `io.verifymcp/probe` — 23 request(s) from 1 address(es), 2026-08-31T23:00:06Z to 2026-09-03T06:12:45Z. - [TAR-Directory-Indexer](/bot/tar-directory-indexer.html) ([json](/bot/tar-directory-indexer.json), [md](/bot/tar-directory-indexer.md)): `TAR-Directory-Indexer/1.0` — 22 request(s) from 1 address(es), 2026-09-01T03:27:40Z to 2026-09-03T03:19:26Z. - [AgentPointsDirectoryEnricher](/bot/agentpointsdirectoryenricher.html) ([json](/bot/agentpointsdirectoryenricher.json), [md](/bot/agentpointsdirectoryenricher.md)): `AgentPointsDirectoryEnricher/1.0 (+https://agentpoints.net)` — 21 request(s) from 1 address(es), 2026-09-02T04:20:57Z to 2026-09-03T04:22:50Z. - [ardcrawl](/bot/ardcrawl.html) ([json](/bot/ardcrawl.json), [md](/bot/ardcrawl.md)): `ardcrawl/0.1 (+https://wellknownhq.com/crawler; ARD catalog census; contact: crawler@wellk` — 21 request(s) from 1 address(es), 2026-09-01T09:41:14Z to 2026-09-03T02:01:40Z. - [lastseen-schema-probe](/bot/lastseen-schema-probe.html) ([json](/bot/lastseen-schema-probe.json), [md](/bot/lastseen-schema-probe.md)): `lastseen-schema-probe/1.0 (+https://lastseen.dev; introspection-only)` — 21 request(s) from 1 address(es), 2026-09-01T06:12:51Z to 2026-09-03T06:07:27Z. - [GoogleOther](/bot/googleother.html) ([json](/bot/googleother.json), [md](/bot/googleother.md)): `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like ` — 20 request(s) from 4 address(es), 2026-09-02T00:10:06Z to 2026-09-03T05:33:51Z. - [GuzzleHttp](/bot/guzzlehttp.html) ([json](/bot/guzzlehttp.json), [md](/bot/guzzlehttp.md)): `GuzzleHttp/7` — 20 request(s) from 6 address(es), 2026-08-31T22:09:48Z to 2026-09-03T05:01:56Z. - [mcp-checker](/bot/mcp-checker.html) ([json](/bot/mcp-checker.json), [md](/bot/mcp-checker.md)): `mcp-checker/1.0` — 20 request(s) from 1 address(es), 2026-09-01T05:31:12Z to 2026-09-02T03:15:32Z. - [meta-externalagent](/bot/meta-externalagent.html) ([json](/bot/meta-externalagent.json), [md](/bot/meta-externalagent.md)): `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/14` — 20 request(s) from 15 address(es), 2026-09-03T00:05:33Z to 2026-09-03T06:24:31Z. - [x402-observatory](/bot/x402-observatory.html) ([json](/bot/x402-observatory.json), [md](/bot/x402-observatory.md)): `x402-observatory/0.2 (+research collector; contact: 300tsb@gmail.com)` — 19 request(s) from 2 address(es), 2026-09-01T06:13:13Z to 2026-09-02T08:39:15Z. - [bingbot](/bot/bingbot.html) ([json](/bot/bingbot.json), [md](/bot/bingbot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bi` — 18 request(s) from 12 address(es), 2026-08-31T20:58:11Z to 2026-09-03T05:19:48Z. - [hultra-link](/bot/hultra-link.html) ([json](/bot/hultra-link.json), [md](/bot/hultra-link.md)): `hultra-link/1.0 (+https://donnees.hultra.link/sondes.md)` — 18 request(s) from 2 address(es), 2026-08-31T22:10:04Z to 2026-09-03T03:20:02Z. - [Orbit-MCP-Registry-IconResolver](/bot/orbit-mcp-registry-iconresolver.html) ([json](/bot/orbit-mcp-registry-iconresolver.json), [md](/bot/orbit-mcp-registry-iconresolver.md)): `Orbit-MCP-Registry-IconResolver/1.0` — 18 request(s) from 4 address(es), 2026-08-31T22:09:49Z to 2026-09-01T00:38:25Z. - [LLMSE](/bot/llmse.html) ([json](/bot/llmse.json), [md](/bot/llmse.md)): `Mozilla/5.0 (compatible; LLMSE/1.7.2; +https://llmse.ai/about)` — 17 request(s) from 1 address(es), 2026-09-03T05:17:24Z to 2026-09-03T05:37:51Z. - [EndpointAudit](/bot/endpointaudit.html) ([json](/bot/endpointaudit.json), [md](/bot/endpointaudit.md)): `EndpointAudit/0.1 (+https://github.com/Zorglub354/endpointaudit)` — 15 request(s) from 3 address(es), 2026-09-01T01:54:55Z to 2026-09-03T01:55:53Z. - [SmitheryBot](/bot/smitherybot.html) ([json](/bot/smitherybot.json), [md](/bot/smitherybot.md)): `SmitheryBot/1.0 (+https://smithery.ai)` — 15 request(s) from 1 address(es), 2026-09-02T09:06:23Z to 2026-09-02T09:07:23Z. - [A2A-Registry-Scanner](/bot/a2a-registry-scanner.html) ([json](/bot/a2a-registry-scanner.json), [md](/bot/a2a-registry-scanner.md)): `A2A-Registry-Scanner/1.0 (Mozilla/5.0 Compatible)` — 14 request(s) from 1 address(es), 2026-09-02T06:43:23Z to 2026-09-03T02:01:47Z. - [agent-guild-scout](/bot/agent-guild-scout.html) ([json](/bot/agent-guild-scout.json), [md](/bot/agent-guild-scout.md)): `agent-guild-scout/1 (+https://agent-guild-5d5r.onrender.com/.well-known/agent-guild.json)` — 14 request(s) from 1 address(es), 2026-09-01T04:15:17Z to 2026-09-03T06:18:04Z. - [mcp-observatory](/bot/mcp-observatory.html) ([json](/bot/mcp-observatory.json), [md](/bot/mcp-observatory.md)): `mcp-observatory/0.1.0 (+https://github.com/yhouta/mcp-observatory; public transparency log` — 14 request(s) from 1 address(es), 2026-09-01T03:08:29Z to 2026-09-03T03:06:42Z. - [mcpgrade-probe](/bot/mcpgrade-probe.html) ([json](/bot/mcpgrade-probe.json), [md](/bot/mcpgrade-probe.md)): `mcpgrade-probe/0.1` — 14 request(s) from 1 address(es), 2026-09-01T07:07:29Z to 2026-09-02T07:08:26Z. - [measure-mcp-schema](/bot/measure-mcp-schema.html) ([json](/bot/measure-mcp-schema.json), [md](/bot/measure-mcp-schema.md)): `measure-mcp-schema/0.1.0` — 14 request(s) from 1 address(es), 2026-09-01T03:33:48Z to 2026-09-03T03:31:06Z. - [QtCreator](/bot/qtcreator.html) ([json](/bot/qtcreator.json), [md](/bot/qtcreator.md)): `QtCreator/20.0.1 (QNetworkAccessManager 6.11.1; OpenMandriva Lx 26.02 (ROME) Rolling; en_U` — 14 request(s) from 13 address(es), 2026-09-01T06:24:35Z to 2026-09-02T12:15:44Z. - [TAR-Discovery](/bot/tar-discovery.html) ([json](/bot/tar-discovery.json), [md](/bot/tar-discovery.md)): `TAR-Discovery/1.0` — 14 request(s) from 1 address(es), 2026-09-01T03:41:21Z to 2026-09-03T03:32:08Z. - [TAR-Health](/bot/tar-health.html) ([json](/bot/tar-health.json), [md](/bot/tar-health.md)): `TAR-Health/1.0` — 14 request(s) from 1 address(es), 2026-09-01T03:41:21Z to 2026-09-03T03:32:08Z. - [utopian-foundry-probe](/bot/utopian-foundry-probe.html) ([json](/bot/utopian-foundry-probe.json), [md](/bot/utopian-foundry-probe.md)): `utopian-foundry-probe/1.0` — 14 request(s) from 1 address(es), 2026-09-01T06:12:38Z to 2026-09-03T06:10:51Z. - [api-forge-mcp-index](/bot/api-forge-mcp-index.html) ([json](/bot/api-forge-mcp-index.json), [md](/bot/api-forge-mcp-index.md)): `api-forge-mcp-index/1.0 (+https://api.temsor.com/mcp/index; iletisim: altyapi@temsor.com)` — 13 request(s) from 1 address(es), 2026-09-01T01:29:04Z to 2026-09-03T01:29:54Z. - [mcp-registry](/bot/mcp-registry.html) ([json](/bot/mcp-registry.json), [md](/bot/mcp-registry.md)): `mcp-registry/1.0` — 13 request(s) from 2 address(es), 2026-08-31T22:02:23Z to 2026-09-02T08:53:27Z. - [mcp-schema-archive](/bot/mcp-schema-archive.html) ([json](/bot/mcp-schema-archive.json), [md](/bot/mcp-schema-archive.md)): `mcp-schema-archive/1.0 (+https://mcp-schema-archive.delfrost42.workers.dev)` — 13 request(s) from 1 address(es), 2026-09-01T03:37:27Z to 2026-09-02T07:33:09Z. - [mcpi](/bot/mcpi.html) ([json](/bot/mcpi.json), [md](/bot/mcpi.md)): `mcpi/probe` — 13 request(s) from 1 address(es), 2026-09-02T23:06:48Z to 2026-09-03T00:31:54Z. - [Mozilla](/bot/mozilla-2.html) ([json](/bot/mozilla-2.json), [md](/bot/mozilla-2.md)): `Mozilla/5.0 (Compatible; Feedsearch Bot; +https://feedsearch.dev)` — 13 request(s) from 4 address(es), 2026-09-01T09:45:16Z to 2026-09-01T15:27:10Z. - [reliability-bureau-spike](/bot/reliability-bureau-spike.html) ([json](/bot/reliability-bureau-spike.json), [md](/bot/reliability-bureau-spike.md)): `reliability-bureau-spike/0.1` — 13 request(s) from 2 address(es), 2026-09-01T09:20:59Z to 2026-09-02T09:27:57Z. - [VerifyMCP-OwnersBot](/bot/verifymcp-ownersbot.html) ([json](/bot/verifymcp-ownersbot.json), [md](/bot/verifymcp-ownersbot.md)): `VerifyMCP-OwnersBot/1.0 (+https://verifymcp.io/docs/build/owners-json)` — 13 request(s) from 1 address(es), 2026-08-31T23:12:20Z to 2026-09-03T04:13:57Z. - [ColonistOne](/bot/colonistone.html) ([json](/bot/colonistone.json), [md](/bot/colonistone.md)): `ColonistOne/1.0 (autonomous agent; review for thecolony.ai; +https://thecolony.ai/u/coloni` — 12 request(s) from 1 address(es), 2026-09-03T05:34:46Z to 2026-09-03T05:34:50Z. - [FreePublicAPIs](/bot/freepublicapis.html) ([json](/bot/freepublicapis.json), [md](/bot/freepublicapis.md)): `Mozilla/5.0 (compatible; FreePublicAPIs/1.0; +https://freepublicapis.com)` — 12 request(s) from 1 address(es), 2026-09-02T04:45:07Z to 2026-09-03T04:50:15Z. - [Googlebot](/bot/googlebot.html) ([json](/bot/googlebot.json), [md](/bot/googlebot.md)): `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` — 12 request(s) from 5 address(es), 2026-09-02T00:11:50Z to 2026-09-03T06:09:41Z. - [SaSameAgentAudit](/bot/sasameagentaudit.html) ([json](/bot/sasameagentaudit.json), [md](/bot/sasameagentaudit.md)): `SaSameAgentAudit/0.1 (+https://live-vps.sasame.online/.well-known/agent-card.json)` — 12 request(s) from 1 address(es), 2026-09-01T01:06:44Z to 2026-09-02T01:40:11Z. - [strand-mcp](/bot/strand-mcp.html) ([json](/bot/strand-mcp.json), [md](/bot/strand-mcp.md)): `strand-mcp/0.1 (+sync)` — 12 request(s) from 1 address(es), 2026-09-02T03:22:45Z to 2026-09-03T03:23:12Z. - [spanly-enrich](/bot/spanly-enrich.html) ([json](/bot/spanly-enrich.json), [md](/bot/spanly-enrich.md)): `spanly-enrich/1.0 (+https://spanly.com)` — 11 request(s) from 1 address(es), 2026-09-01T11:44:27Z to 2026-09-01T11:44:28Z. - [a2a-directory-liveness](/bot/a2a-directory-liveness.html) ([json](/bot/a2a-directory-liveness.json), [md](/bot/a2a-directory-liveness.md)): `a2a-directory-liveness` — 10 request(s) from 1 address(es), 2026-09-01T02:27:27Z to 2026-09-01T15:55:09Z. - [AgentIndexBot](/bot/agentindexbot.html) ([json](/bot/agentindexbot.json), [md](/bot/agentindexbot.md)): `AgentIndexBot/0.1 (+https://agents.traderszone.net; polite ARD crawler)` — 10 request(s) from 1 address(es), 2026-09-01T04:07:21Z to 2026-09-03T04:32:28Z. - [Agentry-Registry](/bot/agentry-registry.html) ([json](/bot/agentry-registry.json), [md](/bot/agentry-registry.md)): `Agentry-Registry/1.0` — 10 request(s) from 1 address(es), 2026-09-01T09:48:46Z to 2026-09-03T06:06:26Z. - [oauth4webapi](/bot/oauth4webapi.html) ([json](/bot/oauth4webapi.json), [md](/bot/oauth4webapi.md)): `oauth4webapi/v3.8.5` — 10 request(s) from 1 address(es), 2026-09-02T09:06:24Z to 2026-09-02T09:07:24Z. - [SolvedEarthPriceBot](/bot/solvedearthpricebot.html) ([json](/bot/solvedearthpricebot.json), [md](/bot/solvedearthpricebot.md)): `Mozilla/5.0 (compatible; SolvedEarthPriceBot/2.0; +https://solved.earth)` — 10 request(s) from 1 address(es), 2026-09-02T02:30:49Z to 2026-09-03T04:14:05Z. - [agent-world-probe](/bot/agent-world-probe.html) ([json](/bot/agent-world-probe.json), [md](/bot/agent-world-probe.md)): `agent-world-probe/py-0.95 (research; MCP census)` — 9 request(s) from 2 address(es), 2026-09-01T08:54:05Z to 2026-09-02T08:04:52Z. - [A2A-Registry-Healthbot](/bot/a2a-registry-healthbot.html) ([json](/bot/a2a-registry-healthbot.json), [md](/bot/a2a-registry-healthbot.md)): `A2A-Registry-Healthbot/1.0 (background-job)` — 8 request(s) from 1 address(es), 2026-09-03T02:01:03Z to 2026-09-03T02:01:47Z. - [jscrawler](/bot/jscrawler.html) ([json](/bot/jscrawler.json), [md](/bot/jscrawler.md)): `Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/)` — 8 request(s) from 1 address(es), 2026-09-03T00:06:47Z to 2026-09-03T00:06:47Z. - [MCPWitness](/bot/mcpwitness.html) ([json](/bot/mcpwitness.json), [md](/bot/mcpwitness.md)): `MCPWitness/1.0 (health probe; +https://mcpwitness.com)` — 8 request(s) from 3 address(es), 2026-09-01T06:30:51Z to 2026-09-03T03:31:38Z. - [pip](/bot/pip.html) ([json](/bot/pip.json), [md](/bot/pip.md)): `pip/24.3.1 {"ci":null,"cpu":"x86_64","implementation":{"name":"CPython","version":"3.11.9"` — 8 request(s) from 1 address(es), 2026-09-01T06:13:45Z to 2026-09-01T18:52:50Z. - [truespar-mcp-registry](/bot/truespar-mcp-registry.html) ([json](/bot/truespar-mcp-registry.json), [md](/bot/truespar-mcp-registry.md)): `truespar-mcp-registry/0.1 (+https://registry.truespar.com)` — 8 request(s) from 1 address(es), 2026-09-01T02:28:06Z to 2026-09-03T02:46:17Z. - [AgentReputationBot](/bot/agentreputationbot.html) ([json](/bot/agentreputationbot.json), [md](/bot/agentreputationbot.md)): `AgentReputationBot/1.0 (+https://agentreputation.dev)` — 7 request(s) from 3 address(es), 2026-09-01T03:47:16Z to 2026-09-03T03:18:55Z. - [merlonix-attestation-verifier](/bot/merlonix-attestation-verifier.html) ([json](/bot/merlonix-attestation-verifier.json), [md](/bot/merlonix-attestation-verifier.md)): `merlonix-attestation-verifier/1` — 7 request(s) from 1 address(es), 2026-09-03T06:00:45Z to 2026-09-03T06:00:45Z. - [PerplexityBot](/bot/perplexitybot.html) ([json](/bot/perplexitybot.json), [md](/bot/perplexitybot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https:/` — 7 request(s) from 5 address(es), 2026-09-02T05:52:18Z to 2026-09-03T06:00:46Z. - [ReactorNetty](/bot/reactornetty.html) ([json](/bot/reactornetty.json), [md](/bot/reactornetty.md)): `ReactorNetty/1.2.1` — 7 request(s) from 4 address(es), 2026-08-31T20:58:14Z to 2026-09-03T05:00:19Z. - [teppi-probe](/bot/teppi-probe.html) ([json](/bot/teppi-probe.json), [md](/bot/teppi-probe.md)): `teppi-probe/0.1.0` — 7 request(s) from 2 address(es), 2026-09-02T02:48:35Z to 2026-09-02T13:17:18Z. - [A2A-Registry-TaskProbe](/bot/a2a-registry-taskprobe.html) ([json](/bot/a2a-registry-taskprobe.json), [md](/bot/a2a-registry-taskprobe.md)): `A2A-Registry-TaskProbe/1.0 (+https://a2aregistry.org)` — 6 request(s) from 3 address(es), 2026-09-01T02:25:59Z to 2026-09-03T02:59:51Z. - [ChainWitness-PR36](/bot/chainwitness-pr36.html) ([json](/bot/chainwitness-pr36.json), [md](/bot/chainwitness-pr36.md)): `ChainWitness-PR36/1.0` — 6 request(s) from 1 address(es), 2026-09-03T04:40:10Z to 2026-09-03T04:40:10Z. - [exaforce-mcprep](/bot/exaforce-mcprep.html) ([json](/bot/exaforce-mcprep.json), [md](/bot/exaforce-mcprep.md)): `exaforce-mcprep/0.1 (MCP server reputation scanner)` — 6 request(s) from 2 address(es), 2026-08-31T22:32:28Z to 2026-09-01T03:23:58Z. - [mcp-drift-monitor](/bot/mcp-drift-monitor.html) ([json](/bot/mcp-drift-monitor.json), [md](/bot/mcp-drift-monitor.md)): `mcp-drift-monitor/0.1 (read-only tool-definition observer)` — 6 request(s) from 1 address(es), 2026-09-01T06:28:10Z to 2026-09-03T00:28:00Z. - [mcp-rugpull-research](/bot/mcp-rugpull-research.html) ([json](/bot/mcp-rugpull-research.json), [md](/bot/mcp-rugpull-research.md)): `mcp-rugpull-research/1.0` — 6 request(s) from 1 address(es), 2026-09-01T16:13:49Z to 2026-09-01T16:13:59Z. - [movanas-registry-snapshot](/bot/movanas-registry-snapshot.html) ([json](/bot/movanas-registry-snapshot.json), [md](/bot/movanas-registry-snapshot.md)): `movanas-registry-snapshot` — 6 request(s) from 1 address(es), 2026-08-31T22:43:36Z to 2026-08-31T23:06:09Z. - [rootz-mcp-registry-prober](/bot/rootz-mcp-registry-prober.html) ([json](/bot/rootz-mcp-registry-prober.json), [md](/bot/rootz-mcp-registry-prober.md)): `rootz-mcp-registry-prober/0.1` — 6 request(s) from 1 address(es), 2026-09-01T09:17:24Z to 2026-09-02T09:17:27Z. - [TelegramBot](/bot/telegrambot.html) ([json](/bot/telegrambot.json), [md](/bot/telegrambot.md)): `TelegramBot (like TwitterBot)` — 6 request(s) from 4 address(es), 2026-09-01T05:44:24Z to 2026-09-02T06:48:07Z. - [a2a-directory-discovery](/bot/a2a-directory-discovery.html) ([json](/bot/a2a-directory-discovery.json), [md](/bot/a2a-directory-discovery.md)): `a2a-directory-discovery` — 5 request(s) from 1 address(es), 2026-09-01T02:27:27Z to 2026-09-01T15:55:09Z. - [mcpqueen-grader](/bot/mcpqueen-grader.html) ([json](/bot/mcpqueen-grader.json), [md](/bot/mcpqueen-grader.md)): `mcpqueen-grader/0.3 (+https://mcpqueen.com)` — 5 request(s) from 1 address(es), 2026-09-01T12:45:45Z to 2026-09-02T07:00:41Z. - [AhrefsBot](/bot/ahrefsbot.html) ([json](/bot/ahrefsbot.json), [md](/bot/ahrefsbot.md)): `Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)` — 4 request(s) from 4 address(es), 2026-09-01T12:22:33Z to 2026-09-01T13:00:44Z. - [axios](/bot/axios.html) ([json](/bot/axios.json), [md](/bot/axios.md)): `axios/1.10.0` — 4 request(s) from 1 address(es), 2026-09-01T07:09:31Z to 2026-09-01T07:43:00Z. - [hlido-a50-probe](/bot/hlido-a50-probe.html) ([json](/bot/hlido-a50-probe.json), [md](/bot/hlido-a50-probe.md)): `hlido-a50-probe/1.0 (+https://hlido.eu/eu-ai-act/)` — 4 request(s) from 1 address(es), 2026-09-03T05:15:32Z to 2026-09-03T05:31:46Z. - [loop-mcp-catalog-fetch](/bot/loop-mcp-catalog-fetch.html) ([json](/bot/loop-mcp-catalog-fetch.json), [md](/bot/loop-mcp-catalog-fetch.md)): `loop-mcp-catalog-fetch/0.1.0` — 4 request(s) from 1 address(es), 2026-09-01T07:49:49Z to 2026-09-01T07:49:49Z. - [mwmbl](/bot/mwmbl.html) ([json](/bot/mwmbl.json), [md](/bot/mwmbl.md)): `mwmbl/0.2.0 (https://github.com/mwmbl/mwmbl/ contact CHANGE_ME@example.com)` — 4 request(s) from 1 address(es), 2026-09-01T06:48:00Z to 2026-09-01T06:48:00Z. - [AgentAlmanac-Snapshot](/bot/agentalmanac-snapshot.html) ([json](/bot/agentalmanac-snapshot.json), [md](/bot/agentalmanac-snapshot.md)): `AgentAlmanac-Snapshot/1.0 (+https://agentalmanac.org)` — 3 request(s) from 1 address(es), 2026-09-02T06:30:42Z to 2026-09-02T06:30:42Z. - [ai-crawler-logs](/bot/ai-crawler-logs.html) ([json](/bot/ai-crawler-logs.json), [md](/bot/ai-crawler-logs.md)): `ai-crawler-logs/1.0.0` — 3 request(s) from 1 address(es), 2026-09-03T00:22:22Z to 2026-09-03T00:22:43Z. - [ai-crawler-robots](/bot/ai-crawler-robots.html) ([json](/bot/ai-crawler-robots.json), [md](/bot/ai-crawler-robots.md)): `ai-crawler-robots/1.0.0` — 3 request(s) from 1 address(es), 2026-09-03T00:22:44Z to 2026-09-03T00:22:47Z. - [ai-crawler-verify](/bot/ai-crawler-verify.html) ([json](/bot/ai-crawler-verify.json), [md](/bot/ai-crawler-verify.md)): `ai-crawler-verify/1.0.0` — 3 request(s) from 2 address(es), 2026-09-01T17:27:19Z to 2026-09-03T00:22:50Z. - [AIVE-MCP-Discover](/bot/aive-mcp-discover.html) ([json](/bot/aive-mcp-discover.json), [md](/bot/aive-mcp-discover.md)): `AIVE-MCP-Discover/1.0 (+https://aive.global/mcp-trust/census; one server/discover POST per` — 3 request(s) from 1 address(es), 2026-09-02T07:31:36Z to 2026-09-02T07:39:50Z. - [AIVE-MCP-EndpointProbe](/bot/aive-mcp-endpointprobe.html) ([json](/bot/aive-mcp-endpointprobe.json), [md](/bot/aive-mcp-endpointprobe.md)): `AIVE-MCP-EndpointProbe/1.0 (+https://github.com/eXaive/aive-ingest; reachability check onl` — 3 request(s) from 1 address(es), 2026-09-01T12:37:31Z to 2026-09-01T12:37:31Z. - [apievangelist-security-probe](/bot/apievangelist-security-probe.html) ([json](/bot/apievangelist-security-probe.json), [md](/bot/apievangelist-security-probe.md)): `Mozilla/5.0 (compatible; apievangelist-security-probe/1.0)` — 3 request(s) from 1 address(es), 2026-09-01T11:59:36Z to 2026-09-01T11:59:36Z. - [MCP-Catalog](/bot/mcp-catalog.html) ([json](/bot/mcp-catalog.json), [md](/bot/mcp-catalog.md)): `Mozilla/5.0 (compatible; MCP-Catalog/1.0)` — 3 request(s) from 1 address(es), 2026-09-01T14:14:28Z to 2026-09-01T14:14:28Z. - [MCP-Stats-Prober](/bot/mcp-stats-prober.html) ([json](/bot/mcp-stats-prober.json), [md](/bot/mcp-stats-prober.md)): `MCP-Stats-Prober/0.1 (+https://github.com/anthropics/mcp-stats)` — 3 request(s) from 3 address(es), 2026-09-01T13:08:09Z to 2026-09-03T01:07:42Z. - [MCPCatalogSync](/bot/mcpcatalogsync.html) ([json](/bot/mcpcatalogsync.json), [md](/bot/mcpcatalogsync.md)): `Mozilla/5.0 (compatible; MCPCatalogSync/1.0)` — 3 request(s) from 3 address(es), 2026-09-01T01:36:35Z to 2026-09-01T01:54:15Z. - [MCPMeter](/bot/mcpmeter.html) ([json](/bot/mcpmeter.json), [md](/bot/mcpmeter.md)): `MCPMeter/1.0 (+https://mcpmeter.dev/about; measurement bot; contact via site)` — 3 request(s) from 1 address(es), 2026-08-31T23:48:38Z to 2026-09-01T11:30:41Z. - [PackageHound](/bot/packagehound.html) ([json](/bot/packagehound.json), [md](/bot/packagehound.md)): `PackageHound/1.0` — 3 request(s) from 3 address(es), 2026-09-01T06:13:51Z to 2026-09-01T17:26:06Z. - [packages.ecosyste.ms](/bot/packages-ecosyste-ms.html) ([json](/bot/packages-ecosyste-ms.json), [md](/bot/packages-ecosyste-ms.md)): `packages.ecosyste.ms` — 3 request(s) from 1 address(es), 2026-09-02T01:01:57Z to 2026-09-03T00:03:35Z. - [ping.blo.gs](/bot/ping-blo-gs.html) ([json](/bot/ping-blo-gs.json), [md](/bot/ping-blo-gs.md)): `ping.blo.gs/2.0` — 3 request(s) from 2 address(es), 2026-09-03T01:21:19Z to 2026-09-03T03:05:22Z. - [A2A-Registry](/bot/a2a-registry.html) ([json](/bot/a2a-registry.json), [md](/bot/a2a-registry.md)): `A2A-Registry/1.0` — 2 request(s) from 2 address(es), 2026-09-01T02:25:59Z to 2026-09-01T09:08:31Z. - [a2a-security-research-crawler](/bot/a2a-security-research-crawler.html) ([json](/bot/a2a-security-research-crawler.json), [md](/bot/a2a-security-research-crawler.md)): `a2a-security-research-crawler/1.0 (academic; contact: security-research)` — 2 request(s) from 1 address(es), 2026-09-02T01:00:09Z to 2026-09-03T01:00:05Z. - [AetherLink-Public-Discovery-Evidence](/bot/aetherlink-public-discovery-evidence.html) ([json](/bot/aetherlink-public-discovery-evidence.json), [md](/bot/aetherlink-public-discovery-evidence.md)): `AetherLink-Public-Discovery-Evidence/1.0` — 2 request(s) from 1 address(es), 2026-09-01T04:00:04Z to 2026-09-01T04:01:16Z. - [AffsignalCrawler](/bot/affsignalcrawler.html) ([json](/bot/affsignalcrawler.json), [md](/bot/affsignalcrawler.md)): `Mozilla/5.0 (compatible; AffsignalCrawler/1.0; +https://affsignal.com/bot)` — 2 request(s) from 2 address(es), 2026-09-02T20:02:10Z to 2026-09-02T20:02:27Z. - [AgentCatalogBot](/bot/agentcatalogbot.html) ([json](/bot/agentcatalogbot.json), [md](/bot/agentcatalogbot.md)): `AgentCatalogBot/1.0 (+https://agentcatalog.com/policy)` — 2 request(s) from 1 address(es), 2026-09-01T13:30:50Z to 2026-09-01T13:33:20Z. - [AzureAI-SearchBot](/bot/azureai-searchbot.html) ([json](/bot/azureai-searchbot.json), [md](/bot/azureai-searchbot.md)): `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;` — 2 request(s) from 1 address(es), 2026-09-02T07:48:11Z to 2026-09-02T07:48:11Z. - [Discordbot](/bot/discordbot.html) ([json](/bot/discordbot.json), [md](/bot/discordbot.md)): `Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)` — 2 request(s) from 2 address(es), 2026-09-02T06:59:22Z to 2026-09-02T07:00:49Z. - [DuckDuckBot](/bot/duckduckbot.html) ([json](/bot/duckduckbot.json), [md](/bot/duckduckbot.md)): `DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)` — 2 request(s) from 1 address(es), 2026-09-02T07:36:46Z to 2026-09-02T07:36:47Z. - [frndOS](/bot/frndos.html) ([json](/bot/frndos.json), [md](/bot/frndos.md)): `frndOS/1 CFNetwork/3860.500.112 Darwin/25.4.0` — 2 request(s) from 2 address(es), 2026-09-01T12:06:12Z to 2026-09-02T09:22:34Z. - [GF-Agent-Toll-Outbound](/bot/gf-agent-toll-outbound.html) ([json](/bot/gf-agent-toll-outbound.json), [md](/bot/gf-agent-toll-outbound.md)): `GF-Agent-Toll-Outbound/1.6.0` — 2 request(s) from 1 address(es), 2026-09-01T02:56:53Z to 2026-09-01T02:56:53Z. - [GF-Agent-Toll-Remediation](/bot/gf-agent-toll-remediation.html) ([json](/bot/gf-agent-toll-remediation.json), [md](/bot/gf-agent-toll-remediation.md)): `GF-Agent-Toll-Remediation/1.0` — 2 request(s) from 1 address(es), 2026-09-01T03:03:03Z to 2026-09-01T03:03:03Z. - [invinoveritas-handshake](/bot/invinoveritas-handshake.html) ([json](/bot/invinoveritas-handshake.json), [md](/bot/invinoveritas-handshake.md)): `invinoveritas-handshake/1.0 (+https://api.babyblueviper.com)` — 2 request(s) from 1 address(es), 2026-09-01T16:15:28Z to 2026-09-01T16:15:28Z. - [llm4agents-cimd-audit](/bot/llm4agents-cimd-audit.html) ([json](/bot/llm4agents-cimd-audit.json), [md](/bot/llm4agents-cimd-audit.md)): `llm4agents-cimd-audit/1.0 (+https://llm4agents.com)` — 2 request(s) from 1 address(es), 2026-09-01T09:12:32Z to 2026-09-01T09:12:32Z. - [mcp-uptime](/bot/mcp-uptime.html) ([json](/bot/mcp-uptime.json), [md](/bot/mcp-uptime.md)): `mcp-uptime/0.3 (+https://mcp-uptime.theopslog.workers.dev)` — 2 request(s) from 1 address(es), 2026-09-01T08:49:41Z to 2026-09-01T08:49:42Z. - [mcpcheck](/bot/mcpcheck.html) ([json](/bot/mcpcheck.json), [md](/bot/mcpcheck.md)): `mcpcheck/0.1 (Plumbline MCP liveness probe; read-only; contact plumbline@airmail.cc)` — 2 request(s) from 1 address(es), 2026-09-02T13:30:55Z to 2026-09-02T13:31:22Z. - [SiteGuardian](/bot/siteguardian.html) ([json](/bot/siteguardian.json), [md](/bot/siteguardian.md)): `SiteGuardian/1.4 (+https://siteguardian.io/bot; EU compliance & uptime monitoring)` — 2 request(s) from 1 address(es), 2026-09-03T06:00:33Z to 2026-09-03T06:00:33Z. - [trustoven-manifest-observer](/bot/trustoven-manifest-observer.html) ([json](/bot/trustoven-manifest-observer.json), [md](/bot/trustoven-manifest-observer.md)): `trustoven-manifest-observer/1.0 (+https://trustoven.com/docs/crawler)` — 2 request(s) from 1 address(es), 2026-09-01T06:22:31Z to 2026-09-01T06:22:31Z. - [Watchpup](/bot/watchpup.html) ([json](/bot/watchpup.json), [md](/bot/watchpup.md)): `Watchpup/1.0 (+https://watchpup.dev; MCP monitor)` — 2 request(s) from 1 address(es), 2026-09-03T05:57:42Z to 2026-09-03T06:12:42Z. - [A2A-Indexer](/bot/a2a-indexer.html) ([json](/bot/a2a-indexer.json), [md](/bot/a2a-indexer.md)): `Mozilla/5.0 (compatible; A2A-Indexer/1.0)` — 1 request(s) from 1 address(es), 2026-09-02T20:21:29Z to 2026-09-02T20:21:29Z. - [AetherLink-Public-Agent-Card-Policy-Check](/bot/aetherlink-public-agent-card-policy-check.html) ([json](/bot/aetherlink-public-agent-card-policy-check.json), [md](/bot/aetherlink-public-agent-card-policy-check.md)): `AetherLink-Public-Agent-Card-Policy-Check/1.0` — 1 request(s) from 1 address(es), 2026-09-01T03:13:23Z to 2026-09-01T03:13:23Z. - [AetherLinkDiscoveryEvidence](/bot/aetherlinkdiscoveryevidence.html) ([json](/bot/aetherlinkdiscoveryevidence.json), [md](/bot/aetherlinkdiscoveryevidence.md)): `AetherLinkDiscoveryEvidence/1.0 (+https://aetherlink.47-85-186-77.sslip.io/llms.txt)` — 1 request(s) from 1 address(es), 2026-09-01T03:30:37Z to 2026-09-01T03:30:37Z. - [AgentSure-MCPScan](/bot/agentsure-mcpscan.html) ([json](/bot/agentsure-mcpscan.json), [md](/bot/agentsure-mcpscan.md)): `Mozilla/5.0 (compatible; AgentSure-MCPScan/0.1; +https://agentsure.tech)` — 1 request(s) from 1 address(es), 2026-09-01T00:24:23Z to 2026-09-01T00:24:23Z. - [Agoragentic-SafeFetch](/bot/agoragentic-safefetch.html) ([json](/bot/agoragentic-safefetch.json), [md](/bot/agoragentic-safefetch.md)): `Agoragentic-SafeFetch/1.0 (+observe-only)` — 1 request(s) from 1 address(es), 2026-09-03T03:40:33Z to 2026-09-03T03:40:33Z. - [apievangelist-domain-probe](/bot/apievangelist-domain-probe.html) ([json](/bot/apievangelist-domain-probe.json), [md](/bot/apievangelist-domain-probe.md)): `apievangelist-domain-probe/1.0` — 1 request(s) from 1 address(es), 2026-09-01T11:59:41Z to 2026-09-01T11:59:41Z. - [Bytespider](/bot/bytespider.html) ([json](/bot/bytespider.json), [md](/bot/bytespider.md)): `Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.` — 1 request(s) from 1 address(es), 2026-09-01T07:20:36Z to 2026-09-01T07:20:36Z. - [CensusBot](/bot/censusbot.html) ([json](/bot/censusbot.json), [md](/bot/censusbot.md)): `CensusBot/0.1 (+https://census.dios.local/about; optout@census.dios.local)` — 1 request(s) from 1 address(es), 2026-09-01T17:00:17Z to 2026-09-01T17:00:17Z. - [Chrome](/bot/chrome.html) ([json](/bot/chrome.json), [md](/bot/chrome.md)): `Chrome Privacy Preserving Prefetch Proxy` — 1 request(s) from 1 address(es), 2026-09-02T05:53:59Z to 2026-09-02T05:53:59Z. - [Conway-Replicatio-r91-strict-a2a-probe](/bot/conway-replicatio-r91-strict-a2a-probe.html) ([json](/bot/conway-replicatio-r91-strict-a2a-probe.json), [md](/bot/conway-replicatio-r91-strict-a2a-probe.md)): `Conway-Replicatio-r91-strict-a2a-probe` — 1 request(s) from 1 address(es), 2026-09-01T04:56:01Z to 2026-09-01T04:56:01Z. - [GraphAdvocate-Outreach](/bot/graphadvocate-outreach.html) ([json](/bot/graphadvocate-outreach.json), [md](/bot/graphadvocate-outreach.md)): `GraphAdvocate-Outreach/1.0 (+https://graphadvocate.com)` — 1 request(s) from 1 address(es), 2026-09-01T09:03:23Z to 2026-09-01T09:03:23Z. - [guild-reachability-probe](/bot/guild-reachability-probe.html) ([json](/bot/guild-reachability-probe.json), [md](/bot/guild-reachability-probe.md)): `guild-reachability-probe/1` — 1 request(s) from 1 address(es), 2026-09-01T08:47:44Z to 2026-09-01T08:47:44Z. - [hhvm-internal](/bot/hhvm-internal.html) ([json](/bot/hhvm-internal.json), [md](/bot/hhvm-internal.md)): `hhvm-internal/thrift-fb-server` — 1 request(s) from 1 address(es), 2026-09-01T17:32:22Z to 2026-09-01T17:32:22Z. - [itinai-importer](/bot/itinai-importer.html) ([json](/bot/itinai-importer.json), [md](/bot/itinai-importer.md)): `itinai-importer/1.0` — 1 request(s) from 1 address(es), 2026-09-02T07:52:09Z to 2026-09-02T07:52:09Z. - [mcp2-research](/bot/mcp2-research.html) ([json](/bot/mcp2-research.json), [md](/bot/mcp2-research.md)): `mcp2-research/1.0 (+https://github.com/dosixx; KHU SIFT Lab)` — 1 request(s) from 1 address(es), 2026-09-02T22:40:49Z to 2026-09-02T22:40:49Z. - [personal-agent-platform-readonly-audit](/bot/personal-agent-platform-readonly-audit.html) ([json](/bot/personal-agent-platform-readonly-audit.json), [md](/bot/personal-agent-platform-readonly-audit.md)): `personal-agent-platform-readonly-audit/1.0` — 1 request(s) from 1 address(es), 2026-09-02T15:08:23Z to 2026-09-02T15:08:23Z. - [repology-linkchecker](/bot/repology-linkchecker.html) ([json](/bot/repology-linkchecker.json), [md](/bot/repology-linkchecker.md)): `repology-linkchecker/1 (+https://repology.org/docs/bots)` — 1 request(s) from 1 address(es), 2026-09-01T07:13:51Z to 2026-09-01T07:13:51Z. - [SaSame-Census-Era-Probe](/bot/sasame-census-era-probe.html) ([json](/bot/sasame-census-era-probe.json), [md](/bot/sasame-census-era-probe.md)): `SaSame-Census-Era-Probe/1.0` — 1 request(s) from 1 address(es), 2026-09-01T01:07:30Z to 2026-09-01T01:07:30Z. - [Telegram](/bot/telegram.html) ([json](/bot/telegram.json), [md](/bot/telegram.md)): `Telegram/34622 CFNetwork/3860.400.51 Darwin/25.3.0` — 1 request(s) from 1 address(es), 2026-09-02T10:15:56Z to 2026-09-02T10:15:56Z. ## If you are cataloguing this API - [/apis.json](/apis.json): APIs.json 1.0 record — identical bytes at [/.well-known/apis.json](/.well-known/apis.json). - [/openapi.json](/openapi.json) and [/openapi.yaml](/openapi.yaml): the same OpenAPI 3.1 document, two serialisations, one generator. Byte-identical copies live at [/.well-known/openapi.json](/.well-known/openapi.json) and [/.well-known/openapi.yaml](/.well-known/openapi.yaml), because several scanners probe those two paths before they read any catalogue. `openapi` is not an IANA-registered well-known name — the registry has `api-catalog` and `security.txt` and no `openapi` — so the root pair stays canonical and each copy says so in a `Link: rel="canonical"` header. - [/asyncapi.json](/asyncapi.json) and [/asyncapi.yaml](/asyncapi.yaml): AsyncAPI 3.0.0 over the message-based half of this host — the five MCP servers, the A2A agent and the cursor-polled change feed — one document, two serialisations, validated against the official AsyncAPI 3.0.0 schema before it is written. Read the second paragraph of `info.description` first: there is no broker, no WebSocket, no webhook and no server-initiated push here, every operation is request/reply over HTTPS, and the only asynchronous framing is the SSE-framed reply the Streamable HTTP transport defines. The REST half stays in `/openapi.json`; the two do not overlap. - [/.well-known/api-onboarding](/.well-known/api-onboarding): how to call it, the limits, what to call first. - [/register](/register) and its machine copy [/register.json](/register.json): access, keys and sign-up — there are none. No account, no API key, no OAuth client registration; every endpoint is public, read-only and CORS-open, and the one endpoint that does take a key (`/hits.json`, the raw request log) says why it is not self-service and points at the public aggregate `/stats.json`. The same document answers at `/signup`, `/api-keys`, `/account/api` and `/developers` and `/developer`, each with a `Link: rel="canonical"` back to `/register`; a POST to any of them is a 405 with reason `not_a_registration_endpoint`, because this host runs no authorization server and must not be mistaken for an RFC 7591 registration endpoint. Its `authentication` and `rate_limits` blocks are generated from the same object as the onboarding document above, so the two cannot disagree. - [/pricing](/pricing) and its machine copy [/pricing.json](/pricing.json): what this host costs. Zero. No plans, no tiers, no trial, no metering, no wallet, no 402 — one price for every caller and nothing to buy, with the offer in schema.org JSON-LD on the page for the pricing crawlers that read that first. The same bytes answer at `/plans`, with a `Link: rel="canonical"` back to `/pricing`. The number is generated from the same function as `/terms.json` (`access.price`) and `/security.json` (`payment`), so the three cannot disagree. - [/.well-known/ai-plugin.json](/.well-known/ai-plugin.json) and [/.well-known/mcp.json](/.well-known/mcp.json): plugin manifest and MCP server record. The MCP one is a CARD, not an endpoint — but POST a JSON-RPC message to it anyway and you get a 307 to the endpoint it describes, method and body preserved (RFC 9110 §15.4.8), with a JSON-RPC error naming that endpoint in the body for clients that do not follow redirects. A per-channel copy redirects to the same channel's endpoint, so attribution survives the hop. This exists because an MCP client POSTed its handshake to the card URL on 2026-09-01 and took a 404; if you publish an MCP server card anywhere a registry will store its `url`, expect the same and consider answering it the same way. The same bytes answer at [/.well-known/mcp](/.well-known/mcp) — extensionless is a spelling, not a second document — and every MCP endpoint on this host also serves a document scoped to ITSELF at `/.well-known/mcp` and `/.well-known/mcp.json`: that server's official registry record, its tools, and its authorization posture (none required), and nothing about the others. Neither path is IANA-registered and MCP defines no well-known discovery document; they exist because registry scanners derive them from the server URLs we publish, and a scanner that derived one took a 404 on three of them at 09:17:24Z. - [/.well-known/api-catalog](/.well-known/api-catalog): RFC 9727 linkset tying all of it together. - [/ai.txt](/ai.txt): text-and-data-mining permissions, in Spawning's grammar, with a byte-identical copy at [/.well-known/ai.txt](/.well-known/ai.txt). Everything here may be mined, indexed and trained on, commercially included; no rights are reserved under EU Directive 2019/790 Art. 4(3), because everything on this host is already CC0. `ai.txt` is a draft convention rather than a standard — not an RFC, not in IANA's well-known registry, and several drafts compete for the name — so treat it as a statement of intent. It is generated from the same crawler list as [/robots.txt](/robots.txt) and cannot disagree with it. - [/terms.html](/terms.html) and [/privacy.html](/privacy.html): terms of use and what one request is logged as — CC0, as-is, no account, no key, no cookie, no rate limit, and a log row that holds a salted hash rather than an address. Machine copies at [/terms.json](/terms.json) and [/privacy.json](/privacy.json); both are declared in [/apis.json](/apis.json) as `TermsOfService` and `PrivacyPolicy`. - [/.well-known/agent-card.json](/.well-known/agent-card.json): A2A v1.0 Agent Card for the agent at `/a2a`, with the legacy [/.well-known/agent.json](/.well-known/agent.json) serving identical bytes. It declares one JSONRPC interface and nine skills; `streaming`, `pushNotifications` and `extendedAgentCard` are all `false` and the endpoint refuses each with the error code the spec assigns rather than pretending. ## Ownership and authorization, for scanners - [/.well-known/owners.json](/.well-known/owners.json), [/mcp/.well-known/owners.json](/mcp/.well-known/owners.json), [/mcp/doctor/.well-known/owners.json](/mcp/doctor/.well-known/owners.json), [/mcp/lint/.well-known/owners.json](/mcp/lint/.well-known/owners.json), [/mcp/netcheck/.well-known/owners.json](/mcp/netcheck/.well-known/owners.json), [/mcp/robots/.well-known/owners.json](/mcp/robots/.well-known/owners.json), [/mcp/triage/.well-known/owners.json](/mcp/triage/.well-known/owners.json): VerifyMCP's ownership document (schema: https://verifymcp.io/schemas/owners.json), identical bytes at every path. Host-level claims every MCP server on this host; each endpoint-level copy claims that one server and nothing else. Three MCP servers, one operator, one host, so both scopes are true. VerifyMCP-OwnersBot/1.0 asks per endpoint — it took a 404 on the `/mcp/triage` and `/mcp/doctor` copies at 2026-09-01T03:12:18Z and 04:12:18Z before they existed — so if you run more than one server behind one hostname, publish one file per endpoint, not just at the root. - The MCP server at `/mcp` requires **no authorization**. There is no `/.well-known/oauth-protected-resource` and no `/.well-known/oauth-authorization-server`, and both are 404 on purpose: RFC 9728 protected-resource metadata "MUST include the authorization_servers field containing at least one authorization server", there is no authorization server here to name, and this host is not one. A 404 at those paths is the specification's own answer for an unauthenticated server, not a gap. **Those 404s now answer in JSON rather than HTML** — request [/.well-known/oauth-protected-resource](/.well-known/oauth-protected-resource), `/.well-known/oauth-authorization-server` or `/.well-known/openid-configuration` (root form, RFC 9728 §3.1 path-insertion form, or `/mcp/.well-known/...`) and the body states the reason, cites the spec, and points at the open endpoint, while the status stays 404. Five named scanners asked for those three paths between 2026-08-31T22:32Z and 2026-09-01T03:23Z and each was handed a human error page; a machine-readable question deserves a machine-readable refusal. The compact version of this paragraph is `x-authorization` in [/.well-known/mcp.json](/.well-known/mcp.json). - [/.well-known/x402](/.well-known/x402): payment discovery, and the answer is **nothing here costs money**. `accepts` is an empty array because there is no paid resource on this host — no wallet, no facilitator, no 402 ever returned, no `PAYMENT-REQUIRED` header ever emitted. Served as **200 rather than 404** because, unlike the OAuth documents above, an honest document exists: "which resources require payment" has the true answer "none", and an empty `accepts` is exactly how x402 spells that. The body also states the negative — `implemented: false`, `payTo: null`, `networks: []` — so nobody can read the file's existence as this host adopting the protocol. Worth knowing if you are being asked for it too: `x402` is **not** a registered well-known URI (absent from the IANA registry, 114 suffixes) and the path appears nowhere in the x402 specification, whose own discovery mechanism is the `bazaar` extension carried inside a real 402 response and catalogued by a facilitator. It is a de-facto convention that payment-discovery crawlers use anyway. AgenstryBot/0.3.0 asked this host for it at 2026-09-01T04:28:51Z with `Accept: application/json` and took a 404; it now gets the document. Same bytes at `/.well-known/x402.json`. - **No `/.well-known/payment-manifest` and no `/.well-known/mpp`** — both 404 by decision, and the 404 now answers in JSON with the reason instead of an HTML page. The same crawler (AgenstryBot/0.3.0) asked for these two in the same second as `x402` above, at 2026-09-01T04:28:51Z. `x402` could be answered honestly and is; these two cannot. The W3C Payment Method Manifest specification **disallows locating that document by guessing a path** (§4.1: it "must be linked via a Link HTTP header" from a payment method identifier URL, and we emit no such header), what it registers with IANA is the *link relation* `payment-method-manifest` rather than a well-known URI, and its format (§2: at most two keys, `default_applications` and `supported_origins`, each **non-empty if present**) has no way to express "this origin is not a payment method" — an empty array is invalid and a populated one is invented. `/.well-known/mpp` has no public specification we could locate at all, and the crawler that asked for it returns 404 on its own. If you are getting the same probe: the file worth serving is `/.well-known/x402` with an empty `accepts`, not a guessed manifest. - [/inference.html](/inference.html) and [/inference.json](/inference.json): **no model runs here**, and `/chat/completions` is 404 by decision. One client swept seven OpenAI-compatible paths in ten seconds on 2026-09-01 (02:49:36Z–02:49:46Z, user-agent exactly `Mozilla/5.0`, three addresses): `/v1/models`, `/openai/v1/models`, `/api/v1/models` and `/models`, then a POST to `/v1/chat/completions`, `/chat/completions` and `/openai/v1/chat/completions` — the standard sweep for an unauthenticated inference gateway. All of them now answer **404 for every method** with a valid OpenAI `ErrorResponse` body (Anthropic's envelope on `/v1/messages` and `/v1/complete`), naming the endpoints that do answer a machine. Two decisions worth copying if you are getting the same probe: an empty `/v1/models` list would be schema-valid and is still refused, because a 200 there asserts the API is implemented and an empty list is what a *broken* gateway returns; and the POSTs used to take a bare `405`, which RFC 9110 §15.5.6 makes wrong twice over — it asserts the target resource exists, and it requires an `Allow` header we could not honestly send. If you *do* run a gateway and that sweep got a 200 without a credential, that is the finding it came for. - **No GraphQL, at any mount point.** `/graphql`, `/api/graphql`, `/query`, `/v1/graphql`, `/graphiql` and `/graphql/console` are **404 for every method, by decision and permanently**. There is no schema here, no resolver and nothing to authenticate to. The body is not the HTML 404 page: it is GraphQL's own envelope — `data: null` with a populated `errors[]`, per GraphQL over HTTP §6.4.2 and GraphQL §7.1.2 — so the client that asked can read the refusal with the parser it already has, and `extensions.code` is `NOT_A_GRAPHQL_ENDPOINT`. Asked for by AgentDisco/1.0 (+https://agentdisco.io/bot), 7 requests on 2026-09-01 between 02:28:31Z and 17:02:40Z. Two decisions worth copying if you are getting the same sweep: a 200 carrying a valid but empty introspection result would be schema-valid and is still refused, because it asserts GraphQL is implemented at this origin and an empty schema is what a broken or permission-stripped gateway returns; and POST used to take a bare zero-byte `405` from the static-asset handler, which RFC 9110 §15.5.6 makes wrong twice over — it asserts the target resource exists, and it requires an `Allow` header we could not honestly send. To a grader that 405 read as a half-built GraphQL API, which is worse than the honest 404. - No `/.well-known/glama.json`: claiming a Glama connector needs an account-bound `glama_claim_...` token, issued only to a signed-in Glama account, and their sign-in is GitHub, Google or Discord OAuth — there is no email route. Their own FAQ says never to publish an email address as ownership proof, and the `maintainers` alternative is marked deprecated in their schema. So no honest document exists for this path: an unclaimed listing is the true state, and this 404 says so. Asked for 18 times in 18 hours by a client that keeps coming back; the answer is still no. - [/security.html](/security.html) and [/security.json](/security.json): what this host runs, and why each path a scanner probes is a 404 rather than hidden. There is no origin server — no Apache, no nginx, no application runtime — so `/server-status`, `/debug`, `/admin`, `/dashboard`, `/_internal`, `/.env`, `/.git/config`, `/config.json`, `/appsettings.json` and `/package.json` are all **404 by construction, and refused on purpose**. `/server-status` in particular would stay refused even on Apache: a truthful scoreboard names in-flight request URLs and the addresses being served, which is our visitors' data and not ours to publish. The aggregate half of that question is already public at [/stats.json](/stats.json). Nothing here returns a soft 404, so a scanner's control probe means what it thinks it means. - [/contact](/contact) · [/legal](/legal) · [/impressum](/impressum) · [/trust](/trust) · [/compliance](/compliance): who runs this and how to reach them, in one place each. One email address, no telephone, no postal address, **no legal entity and no company** — the Impressum names each field German law expects and marks the ones that do not exist rather than inventing them. No certification of any kind is held, and /trust and /compliance say that in their first line. Machine copy: [/contact.json](/contact.json). `/about` and `/about-us` serve [/about.html](/about.html) byte for byte, and `/security` serves [/security.html](/security.html), because that is where three named crawlers asked for them. ## The distinction that matters most Training and AI search are different crawlers with different tokens, and blocking them together is the common expensive mistake. `GPTBot` trains; `OAI-SearchBot` builds the index ChatGPT cites. `ClaudeBot` trains; `Claude-SearchBot` indexes. Google and Apple run no separate AI crawler at all — `Google-Extended` and `Applebot-Extended` are robots.txt control tokens with no user-agent, so blocking `Googlebot` to avoid AI costs you Google Search and achieves nothing else. ## Notes - A user-agent match is a claim, not a proof. Verify against /ip-ranges/ or reverse DNS. - Anthropic, Common Crawl, Meta and ByteDance publish no IP ranges: for those, the user-agent is all there is, and it is trivially spoofable. - Perplexity-User and Bytespider are documented or reported as not governed by robots.txt. A rule for them is a statement of intent; enforcement has to happen at the edge. ## Optional - [Config snippets](/snippet/index.html): nginx, Caddy, Apache, Cloudflare Worker, Python. - [By operator](/operator/index.html) · [By category](/crawler/index.html) · [About and method](/about.html) · [Changelog](/changelog.html) - [Sitemap as markdown](/sitemap.md): every page here, grouped, with links. - [AGENTS.md](/AGENTS.md): the skill file — install, configure, and the seven commands that cover everything this host does. The same bytes at [/agents.md](/agents.md), [/docs/AGENTS.md](/docs/AGENTS.md), [/CLAUDE.md](/CLAUDE.md), [/.cursorrules](/.cursorrules) and [/.cursor/rules](/.cursor/rules). - [Agent Skills](/.well-known/agent-skills/index.json): three installable skills (Cloudflare Agent Skills Discovery 0.2.0) — [identify-ai-crawler](/.well-known/agent-skills/identify-ai-crawler/SKILL.md), [verify-crawler-ip](/.well-known/agent-skills/verify-crawler-ip/SKILL.md), [robots-txt-for-ai](/.well-known/agent-skills/robots-txt-for-ai/SKILL.md). Every command in them runs against this host with no key. - [agents.json](/agents.json): the same operations as agents.json 0.1.0 flows, over /openapi.json. [agent-permissions.json](/.well-known/agent-permissions.json): what an agent may do here (LAS-WG 1.0.0). Nothing here is forbidden except executing script, of which there is none. - Every page on this host also answers as MARKDOWN at the same address with a `.md` extension — `/about.html` is at [/about.md](/about.md), `/contact` is at [/contact.md](/contact.md) — and at `.mdx`, `.html.md` and `.html.mdx`, which are the same bytes. `Accept: text/markdown` on the page itself returns the same document. The HTML page stays canonical and each markdown copy says so in a `Link: rel="canonical"` header. --- # Full index ## AI training crawlers Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today. ### anthropic-ai - Operator: Anthropic - robots.txt token: `anthropic-ai` - User-agent contains: `anthropic-ai` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/anthropic-ai.json A legacy robots.txt token from before Anthropic consolidated on ClaudeBot. It is still widely present in robots.txt files and costs nothing to keep, but it is a control token rather than a bot you will see in logs. **Blocking it costs you:** None. Nothing crawls under this name today; keeping the rule is harmless insurance. ``` User-agent: anthropic-ai Disallow: / ``` ### Applebot-Extended - Operator: Apple - robots.txt token: `Applebot-Extended` - User-agent contains: `(control token only — no crawler)` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://support.apple.com/en-us/119829 - JSON: https://www.pathwren.workers.dev/crawler/applebot-extended.json Apple's counterpart to Google-Extended: a robots.txt token that withdraws consent for Apple Intelligence and Apple foundation-model training, without touching Applebot's search crawl. **Blocking it costs you:** Excluded from Apple Intelligence training. Siri, Spotlight and Safari suggestions are unaffected. ``` User-agent: Applebot-Extended Disallow: / ``` ### Bytespider - Operator: ByteDance - robots.txt token: `Bytespider` - User-agent contains: `Bytespider` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://www.bytespider.net/ - JSON: https://www.pathwren.workers.dev/crawler/bytespider.json ByteDance's crawler, associated with training data collection for Doubao and related models. Repeatedly reported by CDNs and site operators as the highest-volume AI crawler on the web and as inconsistent about robots.txt. **Blocking it costs you:** Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to ask politely in robots.txt. ``` User-agent: Bytespider Disallow: / ``` ### ClaudeBot - Operator: Anthropic - robots.txt token: `ClaudeBot` - User-agent contains: `ClaudeBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claudebot.json Anthropic's bulk crawler, gathering pages that may be used to train Claude models. **Blocking it costs you:** Content excluded from training data for future Claude models. No effect on Claude's ability to fetch a link a user gives it. ``` User-agent: ClaudeBot Disallow: / ``` ### cohere-training-data-crawler - Operator: Cohere - robots.txt token: `cohere-training-data-crawler` - User-agent contains: `cohere-training-data-crawler` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://cohere.com/ - JSON: https://www.pathwren.workers.dev/crawler/cohere-training-data-crawler.json Cohere's separately-named bulk crawler for model training data, split out so consent for training and consent for retrieval can differ. **Blocking it costs you:** Excluded from Cohere model training. ``` User-agent: cohere-training-data-crawler Disallow: / ``` ### Cotoyogi - Operator: ROIS-DS - robots.txt token: `Cotoyogi` - User-agent contains: `Cotoyogi` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://ds.rois.ac.jp/en_center8/en_crawler/ - JSON: https://www.pathwren.workers.dev/crawler/cotoyogi.json A crawler run by ROIS-DS, a Japanese inter-university research organisation, collecting Japanese-language text for AI training. It publishes a crawler page in English and Japanese. **Blocking it costs you:** Your Japanese-language content is left out of an academic training corpus. ``` User-agent: Cotoyogi Disallow: / ``` ### FacebookBot - Operator: Meta - robots.txt token: `FacebookBot` - User-agent contains: `FacebookBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/facebookbot.json Meta's older speech- and language-corpus crawler, largely superseded by meta-externalagent but still listed as a valid robots token. **Blocking it costs you:** Negligible today. Keep the rule; expect little traffic. ``` User-agent: FacebookBot Disallow: / ``` ### Factset_spyderbot - Operator: FactSet - robots.txt token: `Factset_spyderbot` - User-agent contains: `Factset_spyderbot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.factset.com/ai - JSON: https://www.pathwren.workers.dev/crawler/factset-spyderbot.json FactSet's crawler, collecting data used in AI model training for its financial data and analytics products. **Blocking it costs you:** Exclusion from a financial-data vendor's corpus. Relevant mostly to companies whose filings and disclosures are being read. ``` User-agent: Factset_spyderbot Disallow: / ``` ### Google-Extended - Operator: Google - robots.txt token: `Google-Extended` - User-agent contains: `(control token only — no crawler)` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-extended.json Not a crawler. A robots.txt token that tells Google whether pages Googlebot already fetched may be used to train and ground Gemini. You will never see it in an access log; disallowing it changes what Google does with content it fetched under a different name. **Blocking it costs you:** You are excluded from Gemini grounding and Gemini training. Google Search ranking and indexing are explicitly unaffected. This is the cleanest 'no training, keep my search traffic' lever that exists. ``` User-agent: Google-Extended Disallow: / ``` ### GoogleOther - Operator: Google - robots.txt token: `GoogleOther` - User-agent contains: `GoogleOther` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googleother.json A generic fetcher used by Google product teams for one-off crawls and research, including data collection that does not belong to Search. **Blocking it costs you:** No effect on Search indexing. Blocks internal Google research and product fetches. ``` User-agent: GoogleOther Disallow: / ``` ### GoogleOther-Image - Operator: Google - robots.txt token: `GoogleOther-Image` - User-agent contains: `GoogleOther-Image` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googleother-image.json The image variant of GoogleOther: one-off fetches by Google product and research teams that are not Search. It also answers to a GoogleOther group in robots.txt. **Blocking it costs you:** Google teams outside Search stop fetching your images. Image Search itself is unaffected — that is Googlebot-Image. ``` User-agent: GoogleOther-Image Disallow: / ``` ### GoogleOther-Video - Operator: Google - robots.txt token: `GoogleOther-Video` - User-agent contains: `GoogleOther-Video` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googleother-video.json The video variant of GoogleOther, used for internal Google fetches that do not belong to Search. **Blocking it costs you:** No effect on Search or on Google Video search. Blocks internal Google research fetches of your video files. ``` User-agent: GoogleOther-Video Disallow: / ``` ### GPTBot - Operator: OpenAI - robots.txt token: `GPTBot` - User-agent contains: `GPTBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/gptbot.json, 21 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/gptbot.json OpenAI's bulk crawler. Pages it fetches may be used to train future OpenAI foundation models. It is not the bot that puts you in ChatGPT's search results, and blocking it does not remove you from them. **Blocking it costs you:** Your content is excluded from training data for future OpenAI models. No effect on ChatGPT search visibility, on citations, or on links a user pastes into ChatGPT. ``` User-agent: GPTBot Disallow: / ``` ### ICC-Crawler - Operator: NICT - robots.txt token: `ICC-Crawler` - User-agent contains: `ICC-Crawler` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.nict.go.jp/en/ - JSON: https://www.pathwren.workers.dev/crawler/icc-crawler.json Operated by NICT, Japan's national information and communications research institute. The collected data supports AI research and, per the operator, is also provided to third parties including commercial companies. **Blocking it costs you:** You are excluded from a national research corpus and from the commercial redistributions of it. This is a dataset-shaped block: one refusal, many downstream effects. ``` User-agent: ICC-Crawler Disallow: / ``` ### ISSCyberRiskCrawler - Operator: ISS Corporate Solutions - robots.txt token: `ISSCyberRiskCrawler` - User-agent contains: `ISSCyberRiskCrawler` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://iss-cyber.com - JSON: https://www.pathwren.workers.dev/crawler/isscyberriskcrawler.json Crawls in order to train models that score a company's cyber risk. The ai.robots.txt dataset records the operator as not respecting robots.txt; ISS publishes no compliance statement of its own. **Blocking it costs you:** A rule here is a statement of intent. Your organisation's public footprint still gets scored — by a model trained on everybody else. ``` User-agent: ISSCyberRiskCrawler Disallow: / ``` ### Linguee Bot - Operator: Linguee - robots.txt token: `Linguee Bot` - User-agent contains: `Linguee Bot` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://www.linguee.com - JSON: https://www.pathwren.workers.dev/crawler/linguee-bot.json Gathers bilingual text for Linguee's translation corpus and the machine translation trained on it. Recorded in the ai.robots.txt dataset as not respecting robots.txt. **Blocking it costs you:** Multilingual pages stop feeding a translation corpus. If your site is translated, being in it is usually a benefit. ``` User-agent: Linguee Bot Disallow: / ``` ### meta-externalagent - Operator: Meta - robots.txt token: `meta-externalagent` - User-agent contains: `meta-externalagent` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/meta-externalagent.json Meta's AI crawler, gathering training data for Llama and Meta AI. It replaced the older FacebookBot name for this purpose. **Blocking it costs you:** Excluded from Meta AI training. Link previews on Facebook, Instagram and WhatsApp are unaffected — those are a different bot. ``` User-agent: meta-externalagent Disallow: / ``` ### Poseidon Research Crawler - Operator: Poseidon Research - robots.txt token: `Poseidon Research Crawler` - User-agent contains: `Poseidon Research Crawler` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.poseidonresearch.com - JSON: https://www.pathwren.workers.dev/crawler/poseidon-research-crawler.json A crawler run by Poseidon Research, a lab working on interpretability research for AI systems. **Blocking it costs you:** Exclusion from an interpretability research corpus. No published compliance statement. ``` User-agent: Poseidon Research Crawler Disallow: / ``` ### QuillBot - Operator: QuillBot - robots.txt token: `QuillBot` - User-agent contains: `QuillBot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://quillbot.com - JSON: https://www.pathwren.workers.dev/crawler/quillbot.json Operated by QuillBot as part of its writing, paraphrasing and AI-detection products. The dataset also records a second token, quillbot.com, for the same operator. **Blocking it costs you:** Exclusion from QuillBot's corpus. No compliance statement is published, so the rule is a request. ``` User-agent: QuillBot Disallow: / ``` ### Reflectionbot - Operator: Reflection AI - robots.txt token: `Reflectionbot` - User-agent contains: `Reflectionbot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://reflection.ai/ - JSON: https://www.pathwren.workers.dev/crawler/reflectionbot.json An undocumented crawler whose user-agent links to Reflection AI, a company building AI models. The link in the user-agent is the only public statement of purpose that exists. **Blocking it costs you:** Unknown by construction — which is itself the reason some people block it. Nothing user-facing depends on it. ``` User-agent: Reflectionbot Disallow: / ``` ### SBIntuitionsBot - Operator: SB Intuitions - robots.txt token: `SBIntuitionsBot` - User-agent contains: `SBIntuitionsBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.sbintuitions.co.jp/en/bot/ - JSON: https://www.pathwren.workers.dev/crawler/sbintuitionsbot.json SB Intuitions is SoftBank's Japanese LLM lab; this crawler gathers data used in that model development and in information analysis. The operator publishes a dedicated bot page. **Blocking it costs you:** Your content is excluded from a Japanese-language foundation-model corpus. Nothing user-facing changes. ``` User-agent: SBIntuitionsBot Disallow: / ``` ### SemrushBot-OCOB - Operator: Semrush - robots.txt token: `SemrushBot-OCOB` - User-agent contains: `SemrushBot-OCOB` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-ocob.json Semrush's separately-tokenised crawler for its AI content tooling, split out so SEO crawling and AI reuse can be answered differently. **Blocking it costs you:** Exclusion from Semrush's AI corpus, with its SEO crawl unaffected. ``` User-agent: SemrushBot-OCOB Disallow: / ``` ### Sidetrade indexer bot - Operator: Sidetrade - robots.txt token: `Sidetrade indexer bot` - User-agent contains: `Sidetrade indexer bot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.sidetrade.com - JSON: https://www.pathwren.workers.dev/crawler/sidetrade-indexer-bot.json Sidetrade extracts web data for a range of uses including training its AI products for order-to-cash and customer-data work. **Blocking it costs you:** Exclusion from a commercial B2B dataset. The operator publishes no robots.txt statement, so treat the rule as a request rather than a control. ``` User-agent: Sidetrade indexer bot Disallow: / ``` ### TikTokSpider - Operator: ByteDance - robots.txt token: `TikTokSpider` - User-agent contains: `TikTokSpider` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://www.bytespider.net/ - JSON: https://www.pathwren.workers.dev/crawler/tiktokspider.json A second ByteDance crawler identifying with TikTok, collecting page content for the same family of models. **Blocking it costs you:** Little to lose unless TikTok search referral matters to you. ``` User-agent: TikTokSpider Disallow: / ``` ### Webzio-Extended - Operator: Webz.io - robots.txt token: `Webzio-Extended` - User-agent contains: `Webzio-Extended` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://webz.io/blog/machine-learning/ - JSON: https://www.pathwren.workers.dev/crawler/webzio-extended.json Webz.io's opt-out token specifically for AI training reuse, in the pattern Google and Apple established. **Blocking it costs you:** Your content is excluded from the AI-training tier of Webz.io's product while ordinary collection continues. ``` User-agent: Webzio-Extended Disallow: / ``` ### YandexAdditional - Operator: Yandex - robots.txt token: `YandexAdditional` - User-agent contains: `YandexAdditional` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexadditional.json The token that controls whether already-indexed pages may appear in Search with Yandex AI answers. Yandex's table says it makes no indexing requests of its own — it exists so a site can opt out of the generative answer without leaving the index. **Blocking it costs you:** You disappear from Yandex's AI answers while staying in Yandex Search. This is Yandex's equivalent of Google-Extended, and it is the cheap opt-out most people are looking for. ``` User-agent: YandexAdditional Disallow: / ``` ### YandexAdditionalBot - Operator: Yandex - robots.txt token: `YandexAdditionalBot` - User-agent contains: `YandexAdditionalBot` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexadditionalbot.json The second token Yandex publishes for the same AI-answers opt-out. Both names appear in Yandex's own robot list, so a robots.txt that names only one of them is half a policy. **Blocking it costs you:** Same as YandexAdditional: out of Yandex's AI answers, still in Yandex Search. Name both tokens or neither. ``` User-agent: YandexAdditionalBot Disallow: / ``` ## AI search crawlers Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space. ### AIWebIndex - Operator: Lyrenth - robots.txt token: `AIWebIndex` - User-agent contains: `AIWebIndex` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://lyrenth.com/crawler-policy - JSON: https://www.pathwren.workers.dev/crawler/aiwebindex.json Builds an index of public pages and serves them to AI agents as extracted readable text, with attribution and a link back. Lyrenth publishes a crawler policy stating it does not train foundation models on what it collects and that it obeys robots.txt. **Blocking it costs you:** Agents reading through this index stop seeing you — including the attribution and link back that make it a referral rather than a summary. ``` User-agent: AIWebIndex Disallow: / ``` ### Amazonbot - Operator: Amazon - robots.txt token: `Amazonbot` - User-agent contains: `Amazonbot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://developer.amazon.com/amazonbot - JSON: https://www.pathwren.workers.dev/crawler/amazonbot.json Amazon's crawler, feeding Alexa's ability to answer questions from the web and Amazon's own search and assistant products. **Blocking it costs you:** Alexa and Amazon's assistants stop answering from your pages. Verify with reverse DNS to crawl.amazonbot.amazon before trusting the user-agent. ``` User-agent: Amazonbot Disallow: / ``` ### Andibot - Operator: Andi - robots.txt token: `Andibot` - User-agent contains: `Andibot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://andisearch.com/ - JSON: https://www.pathwren.workers.dev/crawler/andibot.json The crawler for Andi, a small generative search assistant that summarises pages rather than listing them. **Blocking it costs you:** You disappear from another assistant's answers. Andi publishes no robots.txt statement. ``` User-agent: Andibot Disallow: / ``` ### Anomura - Operator: Direqt - robots.txt token: `Anomura` - User-agent contains: `Anomura` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://direqt.ai - JSON: https://www.pathwren.workers.dev/crawler/anomura.json Direqt's search crawler. It indexes the sites of Direqt's own publisher customers so their on-site chatbots can answer from them. **Blocking it costs you:** If you are the publisher, this breaks the assistant you put on your own pages. If you are not, it should not be crawling you. ``` User-agent: Anomura Disallow: / ``` ### atlassian-bot - Operator: Atlassian - robots.txt token: `atlassian-bot` - User-agent contains: `atlassian-bot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.atlassian.com/organization-administration/docs/connect-custom-website-to-rovo/ - JSON: https://www.pathwren.workers.dev/crawler/atlassian-bot.json Indexes a website so it can be searched and cited by Rovo, Atlassian's generative assistant inside Jira and Confluence. Atlassian's documentation walks a customer through editing robots.txt for it, which is as close to a compliance statement as this list gets. **Blocking it costs you:** Rovo cannot answer from your public documentation. If your customers live inside Atlassian tools, this is a support-deflection block. ``` User-agent: atlassian-bot Disallow: / ``` ### bedrockbot - Operator: Amazon - robots.txt token: `bedrockbot` - User-agent contains: `bedrockbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html - JSON: https://www.pathwren.workers.dev/crawler/bedrockbot.json The web crawler an AWS customer points at URLs they chose, to build a knowledge base for a Bedrock application. AWS documents that it respects robots.txt and that the user-agent carries a per-customer suffix, so you can allow or refuse one customer's crawl by naming bedrockbot-UUID. **Blocking it costs you:** Companies building retrieval applications on Bedrock cannot include your pages. This is a RAG block, not a training block: nothing is being trained, but nothing can cite you either. ``` User-agent: bedrockbot Disallow: / ``` ### Claude-SearchBot - Operator: Anthropic - robots.txt token: `Claude-SearchBot` - User-agent contains: `Claude-SearchBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claude-searchbot.json Indexes pages so Claude's web search can find and cite them. Separate token from the training crawler, so search visibility and training consent are independent decisions. **Blocking it costs you:** You stop appearing in Claude's search results and citations. ``` User-agent: Claude-SearchBot Disallow: / ``` ### Claude-Web - Operator: Anthropic - robots.txt token: `Claude-Web` - User-agent contains: `Claude-Web` - robots.txt: control token only — no crawler - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claude-web.json An earlier Anthropic token for user-facing web access, superseded by Claude-User and Claude-SearchBot. Kept here because it appears in most published robots.txt templates. **Blocking it costs you:** None in practice. Retain the rule; expect no traffic. ``` User-agent: Claude-Web Disallow: / ``` ### Cloudflare-AutoRAG - Operator: Cloudflare - robots.txt token: `Cloudflare-AutoRAG` - User-agent contains: `Cloudflare-AutoRAG` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.cloudflare.com/ai-search/ - JSON: https://www.pathwren.workers.dev/crawler/cloudflare-autorag.json The crawler behind Cloudflare's AI Search / AutoRAG, which indexes a website into a retrieval index for an application. Cloudflare's own documentation warns that a bot-blocking rule on your zone will also stop this crawler and tells you to allow-list it. **Blocking it costs you:** Applications built on Cloudflare AI Search cannot retrieve your pages. If you are the one building the index over your own site, blocking it breaks your own product. ``` User-agent: Cloudflare-AutoRAG Disallow: / ``` ### DuckAssistBot - Operator: DuckDuckGo - robots.txt token: `DuckAssistBot` - User-agent contains: `DuckAssistBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/ - JSON: https://www.pathwren.workers.dev/crawler/duckassistbot.json Fetches pages so DuckAssist can generate and cite answers inside DuckDuckGo. **Blocking it costs you:** No DuckAssist answers or citations from your site. Ordinary DuckDuckGo results are unaffected. ``` User-agent: DuckAssistBot Disallow: / ``` ### ExaSearchBot - Operator: Exa - robots.txt token: `ExaSearchBot` - User-agent contains: `ExaSearchBot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://exa.ai - JSON: https://www.pathwren.workers.dev/crawler/exasearchbot.json Exa's crawler. It discovers and indexes public pages so they can be retrieved and cited through Exa's search API, which is one of the common retrieval backends behind agent frameworks. **Blocking it costs you:** Agents built on Exa's API stop finding you. Exa publishes no statement about robots.txt compliance, so treat the rule as a request. ``` User-agent: ExaSearchBot Disallow: / ``` ### Google-CloudVertexBot - Operator: Google - robots.txt token: `Google-CloudVertexBot` - User-agent contains: `Google-CloudVertexBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-cloudvertexbot.json Crawls a site on behalf of a Vertex AI Agent Builder customer who is building an agent over that site. It only visits sites the customer has asked it to. **Blocking it costs you:** Third parties can no longer build Vertex AI agents that read your site. Irrelevant to Google Search. ``` User-agent: Google-CloudVertexBot Disallow: / ``` ### KlaviyoAIBot - Operator: Klaviyo - robots.txt token: `KlaviyoAIBot` - User-agent contains: `KlaviyoAIBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://help.klaviyo.com/hc/en-us/articles/40496146232219 - JSON: https://www.pathwren.workers.dev/crawler/klaviyoaibot.json Fetches pages from domains a Klaviyo customer has explicitly connected to their own account, to power Klaviyo's Kai customer agent. It is scoped to connected domains rather than the open web. **Blocking it costs you:** If the connected domain is yours, blocking this breaks the agent you configured. If it is not, this bot should not be reaching you at all. ``` User-agent: KlaviyoAIBot Disallow: / ``` ### Meta-WebIndexer - Operator: Meta - robots.txt token: `Meta-WebIndexer` - User-agent contains: `Meta-WebIndexer` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ - JSON: https://www.pathwren.workers.dev/crawler/meta-webindexer.json Per Meta's crawler documentation, Meta-WebIndexer navigates the web to improve the quality of Meta AI's search results. It is a third Meta token alongside Meta-ExternalAgent and Meta-ExternalFetcher, and the newest of them. **Blocking it costs you:** You leave the index Meta AI answers from across Facebook, Instagram and WhatsApp — the largest assistant install base there is. A robots.txt that names the two older Meta tokens does not cover this one. ``` User-agent: Meta-WebIndexer Disallow: / ``` ### OAI-SearchBot - Operator: OpenAI - robots.txt token: `OAI-SearchBot` - User-agent contains: `OAI-SearchBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/searchbot.json, 35 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/oai-searchbot.json Builds the index ChatGPT search answers from. Content it collects is used for retrieval and citation, not for model training. **Blocking it costs you:** High. Blocking this removes you from ChatGPT search results and from the source links ChatGPT shows. This is the single most expensive block on this list for anyone who wants to be cited by an assistant. ``` User-agent: OAI-SearchBot Disallow: / ``` ### PerplexityBot - Operator: Perplexity - robots.txt token: `PerplexityBot` - User-agent contains: `PerplexityBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://www.perplexity.ai/perplexitybot.json, 8 IPv4 / 0 IPv6) - Docs: https://docs.perplexity.ai/guides/bots - JSON: https://www.pathwren.workers.dev/crawler/perplexitybot.json Builds Perplexity's search index. Perplexity is citation-heavy by product design, so inclusion here converts to referral traffic more directly than most AI surfaces. **Blocking it costs you:** You stop being indexed and cited by Perplexity, and lose the referral clicks its citations produce. ``` User-agent: PerplexityBot Disallow: / ``` ### PhindBot - Operator: Phind - robots.txt token: `PhindBot` - User-agent contains: `PhindBot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.phind.com/ - JSON: https://www.pathwren.workers.dev/crawler/phindbot.json Phind is an answer engine for developers that combines live web search with its own models. This is the crawler behind those answers. **Blocking it costs you:** You stop being cited in answers to technical questions — which, for documentation and reference sites, is the exact audience most worth keeping. ``` User-agent: PhindBot Disallow: / ``` ### QualifiedBot - Operator: Qualified - robots.txt token: `QualifiedBot` - User-agent contains: `QualifiedBot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.qualified.com - JSON: https://www.pathwren.workers.dev/crawler/qualifiedbot.json Analyses a customer's website so Qualified's AI sales chatbots can answer questions about it in context. **Blocking it costs you:** A chatbot on a site that licensed the product loses context. If that site is yours, this block is self-inflicted. ``` User-agent: QualifiedBot Disallow: / ``` ### ShapBot - Operator: Parallel - robots.txt token: `ShapBot` - User-agent contains: `ShapBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.parallel.ai/features/crawler - JSON: https://www.pathwren.workers.dev/crawler/shapbot.json Parallel's crawler. It collects and structures web content to power the search, extraction and deep-research APIs that Parallel sells to agent builders. **Blocking it costs you:** Agents using Parallel's research API lose you as a source. Parallel documents robots.txt compliance, so a rule works. ``` User-agent: ShapBot Disallow: / ``` ### TerraCotta - Operator: Ceramic AI - robots.txt token: `TerraCotta` - User-agent contains: `TerraCotta` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://ceramic.ai/ - JSON: https://www.pathwren.workers.dev/crawler/terracotta.json Ceramic AI's crawler, which indexes public content for a web-scale search API aimed at LLMs and agents. **Blocking it costs you:** You are absent from another agent-facing retrieval index. Ceramic documents that it obeys robots.txt. ``` User-agent: TerraCotta Disallow: / ``` ### YouBot - Operator: You.com - robots.txt token: `YouBot` - User-agent contains: `YouBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://about.you.com/youbot/ - JSON: https://www.pathwren.workers.dev/crawler/youbot.json You.com's crawler, feeding its AI search product and its search API. **Blocking it costs you:** Removal from You.com's index and from answers built on its API. ``` User-agent: YouBot Disallow: / ``` ## User-triggered fetchers Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader. ### ChatGPT Agent - Operator: OpenAI - robots.txt token: `ChatGPT-User` - User-agent contains: `ChatGPT Agent` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/chatgpt-user.json, 204 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/chatgpt-agent.json ChatGPT's agent mode driving a real browser: it navigates and interacts with sites to finish a multi-step task a user gave it. OpenAI governs it with the ChatGPT-User token and the ChatGPT-User prefix list rather than a token of its own, so the robots rule and the address check are the same ones. **Blocking it costs you:** Agentic tasks a user asked for — booking, comparing, filling a form on your site — fail. This is the fetch that ends in a transaction, so it is the most expensive user-triggered block on this list. ``` User-agent: ChatGPT-User Disallow: / ``` ### ChatGPT-User - Operator: OpenAI - robots.txt token: `ChatGPT-User` - User-agent contains: `ChatGPT-User` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://openai.com/chatgpt-user.json, 204 IPv4 / 0 IPv6) - Docs: https://platform.openai.com/docs/bots - JSON: https://www.pathwren.workers.dev/crawler/chatgpt-user.json Fetches a single page at the moment a user or a ChatGPT agent asks for it — a pasted link, a browsing step, an Operator task. One human intent, one request. OpenAI states these fetches are not used for training. **Blocking it costs you:** ChatGPT cannot open your pages when a user explicitly asks it to. The user sees a fetch failure. This is usually the last bot anyone means to block. ``` User-agent: ChatGPT-User Disallow: / ``` ### Claude-User - Operator: Anthropic - robots.txt token: `Claude-User` - User-agent contains: `Claude-User` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://support.anthropic.com/en/articles/8896518 - JSON: https://www.pathwren.workers.dev/crawler/claude-user.json Fetches a page because a Claude user asked Claude to read it, at that moment. **Blocking it costs you:** Claude reports a fetch failure to a user who asked for your page by name. ``` User-agent: Claude-User Disallow: / ``` ### cohere-ai - Operator: Cohere - robots.txt token: `cohere-ai` - User-agent contains: `cohere-ai` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://cohere.com/ - JSON: https://www.pathwren.workers.dev/crawler/cohere-ai.json Cohere's fetcher, used when its assistant products need a page. **Blocking it costs you:** Cohere-powered assistants cannot read your pages on request. ``` User-agent: cohere-ai Disallow: / ``` ### Google-Agent - Operator: Google - robots.txt token: `Google-Agent` - User-agent contains: `Google-Agent` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json, 3 IPv4 / 1 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/google-agent.json Agents hosted on Google infrastructure navigating the web and taking actions on a user's request. Google names one prefix list for it — user-triggered-agents.json — and is separately experimenting with Web Bot Auth under the identity https://agent.bot.goog. **Blocking it costs you:** Google-hosted agents cannot complete a task on your site for a user who asked them to. This is the agentic-commerce fetch: blocking it removes you from what an assistant can actually do rather than from what it can say. ``` User-agent: Google-Agent Disallow: / ``` ### Google-GeminiNotebook - Operator: Google - robots.txt token: `Google-GeminiNotebook` - User-agent contains: `Google-GeminiNotebook` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/google-gemininotebook.json Fetches a URL a Gemini Notebook (formerly NotebookLM) user added as a source to their notebook. The former agent string Google-NotebookLM is documented as supported until August 2026. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** A user who deliberately added your page as a research source gets nothing. This is a citation-shaped fetch, not a training crawl. ``` User-agent: Google-GeminiNotebook Disallow: / ``` ### Google-Pinpoint - Operator: Google - robots.txt token: `Google-Pinpoint` - User-agent contains: `Google-Pinpoint` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/google-pinpoint.json Fetches individual URLs that a Pinpoint user — usually a journalist or researcher — added as a source to their own document collection. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** A researcher who explicitly added your page to a collection cannot load it. ``` User-agent: Google-Pinpoint Disallow: / ``` ### Google-Read-Aloud - Operator: Google - robots.txt token: `Google-Read-Aloud` - User-agent contains: `Google-Read-Aloud` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/google-read-aloud.json Fetches a page so Google can read it out loud with text-to-speech, at the moment a user asks. Formerly google-speakr. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** A reader who asked Google to read your page aloud — often someone using it for accessibility — gets an error instead. ``` User-agent: Google-Read-Aloud Disallow: / ``` ### meta-externalfetcher - Operator: Meta - robots.txt token: `meta-externalfetcher` - User-agent contains: `meta-externalfetcher` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/meta-externalfetcher.json Fetches a page when a Meta AI user asks about a specific link. **Blocking it costs you:** Meta AI cannot read pages users hand it. ``` User-agent: meta-externalfetcher Disallow: / ``` ### MistralAI-User - Operator: Mistral AI - robots.txt token: `MistralAI-User` - User-agent contains: `MistralAI-User` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.mistral.ai/ - JSON: https://www.pathwren.workers.dev/crawler/mistralai-user.json Fetches a page when a Le Chat user asks Mistral's assistant to read it. **Blocking it costs you:** Le Chat cannot open links your readers give it. ``` User-agent: MistralAI-User Disallow: / ``` ### Perplexity-User - Operator: Perplexity - robots.txt token: `Perplexity-User` - User-agent contains: `Perplexity-User` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://www.perplexity.ai/perplexity-user.json, 4 IPv4 / 0 IPv6) - Docs: https://docs.perplexity.ai/guides/bots - JSON: https://www.pathwren.workers.dev/crawler/perplexity-user.json Fetches a page because a Perplexity user asked for it. Perplexity documents that this fetch is user-initiated and is therefore not governed by robots.txt — a robots rule will not stop it, by stated policy. **Blocking it costs you:** Not controllable via robots.txt. If you must stop it, verify by the published IP ranges and block at the edge — and accept that users who ask for your page get an error. ``` User-agent: Perplexity-User Disallow: / ``` ### YandexCalendar - Operator: Yandex - robots.txt token: `YandexCalendar` - User-agent contains: `YandexCalendar` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexcalendar.json Downloads calendar files a user subscribed to. Yandex notes these files are often in directories that are disallowed for indexing, which is why the general rules are not applied. **Blocking it costs you:** Users who subscribed to a calendar you publish stop receiving updates. ``` User-agent: YandexCalendar Disallow: / ``` ## Corpus and dataset builders Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect. ### AI2Bot - Operator: Allen Institute for AI - robots.txt token: `AI2Bot` - User-agent contains: `AI2Bot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://allenai.org/crawler - JSON: https://www.pathwren.workers.dev/crawler/ai2bot.json The Allen Institute's crawler, gathering pages for open research corpora such as Dolma that underpin fully open models like OLMo. **Blocking it costs you:** Excluded from open research datasets. Worth a deliberate decision: this is the category where 'blocking AI' also blocks the open, auditable end of it. ``` User-agent: AI2Bot Disallow: / ``` ### Ai2Bot-Dolma - Operator: Allen Institute for AI - robots.txt token: `Ai2Bot-Dolma` - User-agent contains: `Ai2Bot-Dolma` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://allenai.org/crawler - JSON: https://www.pathwren.workers.dev/crawler/ai2bot-dolma.json The variant of AI2's crawler named for the Dolma corpus specifically. **Blocking it costs you:** Same as AI2Bot: exclusion from an open, published training corpus. ``` User-agent: Ai2Bot-Dolma Disallow: / ``` ### aiHitBot - Operator: aiHit - robots.txt token: `aiHitBot` - User-agent contains: `aiHitBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.aihitdata.com/about - JSON: https://www.pathwren.workers.dev/crawler/aihitbot.json aiHit's automated collector, building a company dataset from public company websites. **Blocking it costs you:** Your company record in a B2B dataset goes stale. Documented as respecting robots.txt, so the rule works. ``` User-agent: aiHitBot Disallow: / ``` ### AwarioRssBot - Operator: Awario - robots.txt token: `AwarioRssBot` - User-agent contains: `AwarioRssBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://awario.com/bots.html - JSON: https://www.pathwren.workers.dev/crawler/awariorssbot.json The feed-reading half of Awario's pair, documented on the same page and under the same crawl-rate policy. **Blocking it costs you:** Your RSS updates stop reaching Awario's monitoring. Block both tokens or neither. ``` User-agent: AwarioRssBot Disallow: / ``` ### AwarioSmartBot - Operator: Awario - robots.txt token: `AwarioSmartBot` - User-agent contains: `AwarioSmartBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://awario.com/bots.html - JSON: https://www.pathwren.workers.dev/crawler/awariosmartbot.json Awario's brand-monitoring crawler. It documents one request per three seconds, honours Crawl-delay, and states it does not use consecutive IP blocks so identification is by user-agent only. **Blocking it costs you:** Mentions of brands on your pages stop being surfaced to the people monitoring them — including, quite possibly, your own. ``` User-agent: AwarioSmartBot Disallow: / ``` ### CCBot - Operator: Common Crawl - robots.txt token: `CCBot` - User-agent contains: `CCBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://index.commoncrawl.org/ccbot.json, 4 IPv4 / 1 IPv6) - Docs: https://commoncrawl.org/ccbot - JSON: https://www.pathwren.workers.dev/crawler/ccbot.json Common Crawl's corpus builder. It trains nothing itself, but its archive is an input to most open and many closed LLM training sets, which makes it the highest-leverage single entry on this list. **Blocking it costs you:** Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but only going forward. Existing snapshots are permanent and blocking today does not retract them. ``` User-agent: CCBot Disallow: / ``` ### Diffbot - Operator: Diffbot - robots.txt token: `Diffbot` - User-agent contains: `Diffbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.diffbot.com/ - JSON: https://www.pathwren.workers.dev/crawler/diffbot.json Extracts structured records from pages to build a commercial knowledge graph that is resold and used for retrieval and training. **Blocking it costs you:** Your facts stop entering a widely-licensed knowledge graph. Whether that is a loss depends on whether you want to be a machine-readable entity. ``` User-agent: Diffbot Disallow: / ``` ### EchoboxBot - Operator: Echobox - robots.txt token: `EchoboxBot` - User-agent contains: `EchoboxBot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://echobox.com - JSON: https://www.pathwren.workers.dev/crawler/echoboxbot.json Collects data supporting Echobox's AI-driven social and email distribution products, which publishers use to schedule and target their own content. **Blocking it costs you:** Publishers using Echobox get worse scheduling decisions about your articles. No compliance statement is published. ``` User-agent: EchoboxBot Disallow: / ``` ### ImagesiftBot - Operator: Hive AI - robots.txt token: `ImagesiftBot` - User-agent contains: `ImagesiftBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://imagesift.com/about - JSON: https://www.pathwren.workers.dev/crawler/imagesiftbot.json Crawls images for Hive AI's reverse-image and dataset products. Image-heavy sites see this one long before they see the text crawlers. **Blocking it costs you:** Your images stop entering an image dataset and reverse-image index. ``` User-agent: ImagesiftBot Disallow: / ``` ### img2dataset - Operator: LAION / img2dataset - robots.txt token: `img2dataset` - User-agent contains: `img2dataset` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://github.com/rom1504/img2dataset - JSON: https://www.pathwren.workers.dev/crawler/img2dataset.json The tool used to turn image-URL lists such as LAION's into downloaded training sets. It is run by whoever is building a dataset, not by a single operator. **Blocking it costs you:** Your images are skipped when someone materialises an image-text dataset that references them. ``` User-agent: img2dataset Disallow: / ``` ### LAIONDownloader - Operator: LAION / img2dataset - robots.txt token: `LAIONDownloader` - User-agent contains: `LAIONDownloader` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: no published verification method - Docs: https://laion.ai/faq/ - JSON: https://www.pathwren.workers.dev/crawler/laiondownloader.json LAION's downloader, used to materialise the image and text datasets the non-profit publishes for machine-learning research. LAION's own FAQ is the source for its robots.txt position. **Blocking it costs you:** Your media is skipped when an open research dataset is built from URL lists. Once a dataset is published, a later block does not remove you from it. ``` User-agent: LAIONDownloader Disallow: / ``` ### omgili - Operator: Webz.io - robots.txt token: `omgili` - User-agent contains: `omgili` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://webz.io/blog/machine-learning/ - JSON: https://www.pathwren.workers.dev/crawler/omgili.json The older robots token for the same Webz.io collection, still honoured and still worth listing. **Blocking it costs you:** Same as omgilibot. ``` User-agent: omgili Disallow: / ``` ### omgilibot - Operator: Webz.io - robots.txt token: `omgilibot` - User-agent contains: `omgilibot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://webz.io/blog/machine-learning/ - JSON: https://www.pathwren.workers.dev/crawler/omgilibot.json Webz.io's crawler, collecting web and forum text sold as datasets, including to model builders. **Blocking it costs you:** Exclusion from a commercial dataset resold to third parties. ``` User-agent: omgilibot Disallow: / ``` ### Panscient - Operator: Panscient - robots.txt token: `panscient.com` - User-agent contains: `panscient.com` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://panscient.com/faq.htm - JSON: https://www.pathwren.workers.dev/crawler/panscient.json Compiles structured data about businesses and business professionals using machine learning. Panscient's FAQ states it obeys robots.txt. **Blocking it costs you:** Your company pages stop feeding a business-data product. No effect on search or assistants. ``` User-agent: panscient.com Disallow: / ``` ### Thinkbot - Operator: Thinkbot - robots.txt token: `Thinkbot` - User-agent contains: `Thinkbot` - robots.txt: compliance disputed - Verify by: no published verification method - Docs: https://www.thinkbot.agency - JSON: https://www.pathwren.workers.dev/crawler/thinkbot.json Collects pages for analysis of how sites are adopting AI and automation. The ai.robots.txt dataset records the operator as not respecting robots.txt. **Blocking it costs you:** Exclusion from a market-research dataset. Expect to enforce this at the edge rather than in robots.txt. ``` User-agent: Thinkbot Disallow: / ``` ### VelenPublicWebCrawler - Operator: Hunter (Velen) - robots.txt token: `VelenPublicWebCrawler` - User-agent contains: `VelenPublicWebCrawler` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://velen.io/ - JSON: https://www.pathwren.workers.dev/crawler/velenpublicwebcrawler.json Hunter's crawler, written in Go, building business datasets and machine-learning models from public pages. Its page states it follows robots.txt and meta directives and never fetches more than one page every two seconds. **Blocking it costs you:** Your company pages stop feeding a B2B contact and company dataset. The crawl rate it documents makes this one of the cheapest visitors to simply allow. ``` User-agent: VelenPublicWebCrawler Disallow: / ``` ### YaK - Operator: Meltwater - robots.txt token: `YaK` - User-agent contains: `YaK` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.meltwater.com/en/suite/consumer-intelligence - JSON: https://www.pathwren.workers.dev/crawler/yak.json Meltwater's crawler, feeding the live data stream behind its media-monitoring and consumer-intelligence suite. **Blocking it costs you:** Your content stops appearing in Meltwater's media monitoring — which is how PR teams find out you were mentioned. Some publishers want to be in it. ``` User-agent: YaK Disallow: / ``` ## Search engines Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block. ### Applebot - Operator: Apple - robots.txt token: `Applebot` - User-agent contains: `Applebot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://search.developer.apple.com/applebot.json, 33 IPv4 / 0 IPv6) - Docs: https://support.apple.com/en-us/119829 - JSON: https://www.pathwren.workers.dev/crawler/applebot.json Powers Siri, Spotlight and Safari suggestions. Blocking it is a search decision, not an AI decision — the AI decision has its own token. **Blocking it costs you:** You disappear from Siri, Spotlight and Safari search suggestions across Apple's install base. ``` User-agent: Applebot Disallow: / ``` ### Baiduspider - Operator: Baidu - robots.txt token: `Baiduspider` - User-agent contains: `Baiduspider` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://help.baidu.com/question?prod_id=99&class=0&id=3001 - JSON: https://www.pathwren.workers.dev/crawler/baiduspider.json Baidu's search crawler, and the ingest path for Baidu's Ernie-backed answers. **Blocking it costs you:** Removal from Baidu Search, which matters only if you want Chinese-language traffic. ``` User-agent: Baiduspider Disallow: / ``` ### bingbot - Operator: Microsoft - robots.txt token: `bingbot` - User-agent contains: `bingbot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://www.bing.com/toolbox/bingbot.json, 28 IPv4 / 0 IPv6) - Docs: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 - JSON: https://www.pathwren.workers.dev/crawler/bingbot.json Bing's only crawler, and therefore also the crawler behind Microsoft Copilot's grounding. Microsoft's documented way to keep search indexing while refusing generative reuse is the nocache / noarchive robots meta directive, not a separate user-agent. **Blocking it costs you:** Very high and very wide: Bing, Copilot, DuckDuckGo and several assistants that resell Bing's index all lose you at once. Use nocache/noarchive rather than blocking. ``` User-agent: bingbot Disallow: / ``` ### DuckDuckBot - Operator: DuckDuckGo - robots.txt token: `DuckDuckBot` - User-agent contains: `DuckDuckBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://duckduckgo.com/duckduckbot.json, 486 IPv4 / 0 IPv6) - Docs: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ - JSON: https://www.pathwren.workers.dev/crawler/duckduckbot.json DuckDuckGo's own crawler. Note that the bulk of DuckDuckGo's web results come from Bing, so blocking bingbot removes you from DuckDuckGo whether or not you allow this one. **Blocking it costs you:** Limited on its own; the real DuckDuckGo lever is bingbot. ``` User-agent: DuckDuckBot Disallow: / ``` ### Googlebot - Operator: Google - robots.txt token: `Googlebot` - User-agent contains: `Googlebot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 170 IPv4 / 147 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot.json The classic search crawler. It is also the crawler behind AI Overviews: Google does not run a separate bot for them, which is why the only AI opt-out is the Google-Extended token and not a Googlebot block. **Blocking it costs you:** Total. You leave Google Search. Never block this to avoid AI use; use Google-Extended instead. ``` User-agent: Googlebot Disallow: / ``` ### Googlebot-Image - Operator: Google - robots.txt token: `Googlebot-Image` - User-agent contains: `Googlebot-Image` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 170 IPv4 / 147 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot-image.json Image indexing for Google Images. A separate token so you can leave images out of search without leaving search. **Blocking it costs you:** Your images stop appearing in Google Images. ``` User-agent: Googlebot-Image Disallow: / ``` ### Googlebot-News - Operator: Google - robots.txt token: `Googlebot-News` - User-agent contains: `Googlebot-News` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 170 IPv4 / 147 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot-news.json A robots.txt token controlling inclusion in Google News. It does not have its own user-agent string; the fetch arrives as Googlebot. **Blocking it costs you:** Removal from Google News, with normal Search unaffected. ``` User-agent: Googlebot-News Disallow: / ``` ### Googlebot-Video - Operator: Google - robots.txt token: `Googlebot-Video` - User-agent contains: `Googlebot-Video` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/googlebot.json, 170 IPv4 / 147 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers - JSON: https://www.pathwren.workers.dev/crawler/googlebot-video.json The video half of Googlebot. It crawls video files and the pages around them for Google Video search, and it is matched by a robots.txt group for Googlebot as well as by its own token. **Blocking it costs you:** Your videos leave Google video search. A rule for Googlebot already covers it, so blocking this token alone is usually a mistake of precision rather than of intent. ``` User-agent: Googlebot-Video Disallow: / ``` ### Kagibot - Operator: Kagi - robots.txt token: `Kagibot` - User-agent contains: `Kagibot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://kagi.com/bot - JSON: https://www.pathwren.workers.dev/crawler/kagibot.json The crawler for Kagi, a paid, ad-free search engine with its own index and its own assistant. **Blocking it costs you:** You leave Kagi's index. Kagi's users are paying to search and skew technical; per visitor this is an expensive block. ``` User-agent: Kagibot Disallow: / ``` ### MojeekBot - Operator: Mojeek - robots.txt token: `MojeekBot` - User-agent contains: `MojeekBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.mojeek.com/bot.html - JSON: https://www.pathwren.workers.dev/crawler/mojeekbot.json Mojeek's crawler. Mojeek runs one of the few genuinely independent web indexes — not a front end over Bing or Google — so it is one of the few blocks that removes you from an index nobody else can put you back into. Its documentation states it obeys the first record whose User-Agent contains MojeekBot, falling back to *. **Blocking it costs you:** You leave an independent index that other privacy-focused search products draw on. Small traffic, disproportionate long-term cost to web plurality. ``` User-agent: MojeekBot Disallow: / ``` ### PetalBot - Operator: Huawei - robots.txt token: `PetalBot` - User-agent contains: `PetalBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://aspiegel.com/petalbot - JSON: https://www.pathwren.workers.dev/crawler/petalbot.json Huawei's crawler for Petal Search, shipped as the default search on Huawei devices. **Blocking it costs you:** Removal from Petal Search. Frequently blocked for volume rather than for policy. ``` User-agent: PetalBot Disallow: / ``` ### Pinterestbot - Operator: Pinterest - robots.txt token: `Pinterestbot` - User-agent contains: `Pinterestbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://help.pinterest.com/en/business/article/pinterest-crawler - JSON: https://www.pathwren.workers.dev/crawler/pinterestbot.json Pinterest's crawler. It indexes pages so people can find them on Pinterest and re-reads product pages to keep price and title on a Pin current. Pinterest states that content it crawls is not used to train their Canvas image generation model. **Blocking it costs you:** Pins pointing at your site go stale — wrong prices, dead links — and new content stops being indexed. For a retailer this is one of the more expensive blocks on the list. ``` User-agent: Pinterestbot Disallow: / ``` ### Qwantbot - Operator: Qwant - robots.txt token: `Qwantbot` - User-agent contains: `Qwantbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://help.qwant.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/qwantbot.json Qwant's crawler. Qwant documents that the string Qwantbot always appears in its user-agents whatever the crawler version, which is what makes a substring match safe here. **Blocking it costs you:** You leave the index behind Qwant, the French privacy-focused engine, and the products that federate it. ``` User-agent: Qwantbot Disallow: / ``` ### Qwantbot-news - Operator: Qwant - robots.txt token: `Qwantbot-news` - User-agent contains: `Qwantbot-news` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://help.qwant.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/qwantbot-news.json The news variant of Qwant's crawler, documented alongside the main one and carrying the same Qwantbot substring. **Blocking it costs you:** Your articles stop appearing in Qwant News. A rule for Qwantbot as a substring already catches both. ``` User-agent: Qwantbot-news Disallow: / ``` ### SeznamBot - Operator: Seznam - robots.txt token: `SeznamBot` - User-agent contains: `SeznamBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://napoveda.seznam.cz/en/seznamzbozi/subject-matter-crawler/ - JSON: https://www.pathwren.workers.dev/crawler/seznambot.json Seznam's crawler — the dominant search engine in the Czech Republic and one of the few national engines with its own index. **Blocking it costs you:** Removal from Seznam. Also removes you from its IndexNow endpoint's usefulness. ``` User-agent: SeznamBot Disallow: / ``` ### Storebot-Google - Operator: Google - robots.txt token: `Storebot-Google` - User-agent contains: `Storebot-Google` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/storebot-google.json Checks shopping and checkout flows for Google's shopping surfaces. **Blocking it costs you:** Product listings may lose shopping-specific enrichment. Irrelevant to non-commerce sites. ``` User-agent: Storebot-Google Disallow: / ``` ### Timpibot - Operator: Timpi - robots.txt token: `Timpibot` - User-agent contains: `Timpibot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://timpi.io/ - JSON: https://www.pathwren.workers.dev/crawler/timpibot.json A distributed crawler building an independent search index outside the Google/Bing duopoly. **Blocking it costs you:** Absence from a small independent index. ``` User-agent: Timpibot Disallow: / ``` ### YandexBlogs - Operator: Yandex - robots.txt token: `YandexBlogs` - User-agent contains: `YandexBlogs` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexblogs.json Yandex's blog-search robot; it indexes post comments as well as posts. **Blocking it costs you:** Comment threads and blog posts stop being findable through Yandex blog search. ``` User-agent: YandexBlogs Disallow: / ``` ### YandexBot - Operator: Yandex - robots.txt token: `YandexBot` - User-agent contains: `YandexBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html - JSON: https://www.pathwren.workers.dev/crawler/yandexbot.json Yandex's search crawler, which also feeds Alice and Yandex's generative answers. **Blocking it costs you:** Removal from Yandex Search. Verify with reverse DNS to a yandex.ru, yandex.net or yandex.com host — YandexBot is among the most-spoofed user-agents there is. ``` User-agent: YandexBot Disallow: / ``` ### YandexComBot - Operator: Yandex - robots.txt token: `YandexComBot` - User-agent contains: `YandexComBot` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexcombot.json Indexes content for Yandex search in languages other than Russian. Yandex documents that it can index content when there is no explicit robot-specific restriction — a * group is not one. **Blocking it costs you:** You leave Yandex's non-Russian index. A rule naming this token is the only one that works on it. ``` User-agent: YandexComBot Disallow: / ``` ### YandexFavicons - Operator: Yandex - robots.txt token: `YandexFavicons` - User-agent contains: `YandexFavicons` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexfavicons.json Downloads your favicon so Yandex can show it beside your result. Documented as not taking the general robots.txt rules into account. **Blocking it costs you:** Your results in Yandex lose their icon. Cosmetic, and a * rule will not achieve it anyway. ``` User-agent: YandexFavicons Disallow: / ``` ### YandexImages - Operator: Yandex - robots.txt token: `YandexImages` - User-agent contains: `YandexImages` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandeximages.json Indexes images for Yandex Images. Yandex's robot table marks it as taking the general robots.txt rules into account. **Blocking it costs you:** Your images leave Yandex Images, which is a large share of image search in Russian-speaking markets. ``` User-agent: YandexImages Disallow: / ``` ### YandexMarket - Operator: Yandex - robots.txt token: `YandexMarket` - User-agent contains: `YandexMarket` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexmarket.json The robot behind Yandex Market, Yandex's shopping comparison service. Version 1.0 is documented as obeying the general rules; version 2.0 is documented as not. **Blocking it costs you:** Your products stop being listed and priced in Yandex Market. For a retailer in that market this is a revenue block, not a bandwidth one. ``` User-agent: YandexMarket Disallow: / ``` ### YandexMedia - Operator: Yandex - robots.txt token: `YandexMedia` - User-agent contains: `YandexMedia` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexmedia.json Indexes multimedia data for Yandex. Takes the general robots.txt rules into account. **Blocking it costs you:** Your multimedia content stops appearing in Yandex's media surfaces. ``` User-agent: YandexMedia Disallow: / ``` ### YandexMobileBot - Operator: Yandex - robots.txt token: `YandexMobileBot` - User-agent contains: `YandexMobileBot` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexmobilebot.json Decides whether a page's layout is suitable for mobile devices. Yandex's table marks it as NOT taking the general robots.txt rules into account, so a * group does not stop it — a group named YandexMobileBot does. **Blocking it costs you:** Yandex loses its mobile-friendliness signal for your pages, which affects how they are ranked and rendered on phones. ``` User-agent: YandexMobileBot Disallow: / ``` ### YandexRenderResourcesBot - Operator: Yandex - robots.txt token: `YandexRenderResourcesBot` - User-agent contains: `YandexRenderResourcesBot` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexrenderresourcesbot.json Loads the CSS, JavaScript and images Yandex needs to render a page. Yandex documents the exact rule: it ignores robots.txt for a resource when the HTML page using it is allowed, and does not fetch the resource when that page is disallowed. **Blocking it costs you:** Yandex renders your pages without their stylesheets or scripts and ranks what it sees. This is the classic accidental self-inflicted ranking loss. ``` User-agent: YandexRenderResourcesBot Disallow: / ``` ### YandexVideo - Operator: Yandex - robots.txt token: `YandexVideo` - User-agent contains: `YandexVideo` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexvideo.json Indexes video for Yandex video search. Obeys the general robots.txt rules per Yandex's own table. **Blocking it costs you:** Removal from Yandex video search. Note that a second robot, YandexVideoParser, does the same job and is documented as NOT taking the general rules into account. ``` User-agent: YandexVideo Disallow: / ``` ### Yeti - Operator: Naver - robots.txt token: `Yeti` - User-agent contains: `Yeti` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://searchadvisor.naver.com/guide/seo-basic-crawl - JSON: https://www.pathwren.workers.dev/crawler/yeti.json Naver's crawler. Naver is South Korea's largest search portal and runs its own index and its own generative answers. **Blocking it costs you:** Removal from Naver, which is most of Korean search. ``` User-agent: Yeti Disallow: / ``` ## SEO and backlink crawlers Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth. ### AhrefsBot - Operator: Ahrefs - robots.txt token: `AhrefsBot` - User-agent contains: `AhrefsBot` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://api.ahrefs.com/v3/public/crawler-ip-ranges, 81 IPv4 / 0 IPv6) - Docs: https://ahrefs.com/robot - JSON: https://www.pathwren.workers.dev/crawler/ahrefsbot.json Ahrefs' backlink crawler, and one of the largest non-search crawlers on the web by request volume. **Blocking it costs you:** No user-facing effect. Ahrefs honours Crawl-delay, so rate-limiting is usually better than blocking. ``` User-agent: AhrefsBot Disallow: / ``` ### AhrefsSiteAudit - Operator: Ahrefs - robots.txt token: `AhrefsSiteAudit` - User-agent contains: `AhrefsSiteAudit` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://api.ahrefs.com/v3/public/crawler-ip-ranges, 81 IPv4 / 0 IPv6) - Docs: https://ahrefs.com/robot - JSON: https://www.pathwren.workers.dev/crawler/ahrefssiteaudit.json Ahrefs' site-audit crawler, separate from AhrefsBot. Ahrefs documents that it obeys robots.txt by default, and that a verified site owner can ask for it to be allowed to ignore robots.txt on their own site so the audit can see disallowed sections. **Blocking it costs you:** Site owners auditing your domain get an incomplete report. Blocking it saves bandwidth and costs you nothing in search. ``` User-agent: AhrefsSiteAudit Disallow: / ``` ### Barkrowler - Operator: Babbar - robots.txt token: `barkrowler` - User-agent contains: `barkrowler` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://babbar.tech/crawler - JSON: https://www.pathwren.workers.dev/crawler/barkrowler.json Babbar's crawler, which builds the link graph behind their French-market SEO tooling. **Blocking it costs you:** You leave Babbar's index. No effect on search or assistants. ``` User-agent: barkrowler Disallow: / ``` ### DataForSeoBot - Operator: DataForSEO - robots.txt token: `DataForSeoBot` - User-agent contains: `DataForSeoBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://dataforseo.com/dataforseo-bot - JSON: https://www.pathwren.workers.dev/crawler/dataforseobot.json Builds the backlink and SERP datasets DataForSEO resells through its API, so one crawl reaches many downstream tools. **Blocking it costs you:** You leave a dataset that a long tail of SEO products is built on. No user-facing effect. ``` User-agent: DataForSeoBot Disallow: / ``` ### DotBot - Operator: Moz - robots.txt token: `dotbot` - User-agent contains: `dotbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://moz.com/help/moz-procedures/crawlers/dotbot - JSON: https://www.pathwren.workers.dev/crawler/dotbot.json Moz's crawler for Link Explorer. Moz documents that it respects robots.txt and that dotbot is the token to name. **Blocking it costs you:** You leave Moz's link index, so Domain Authority and link reports about your site get thinner. Nothing a reader or an assistant sees changes. ``` User-agent: dotbot Disallow: / ``` ### MJ12bot - Operator: Majestic - robots.txt token: `MJ12bot` - User-agent contains: `MJ12bot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://mj12bot.com/ - JSON: https://www.pathwren.workers.dev/crawler/mj12bot.json Majestic's link-graph crawler, run as a distributed community project. Majestic states plainly that it cannot restrict the bot to a fixed set of addresses, and offers a pre-arranged ident string in the request headers instead. **Blocking it costs you:** You leave the Majestic backlink index. No search or AI effect. It supports Crawl-delay, which is usually the better answer than a block. ``` User-agent: MJ12bot Disallow: / ``` ### rogerbot - Operator: Moz - robots.txt token: `rogerbot` - User-agent contains: `rogerbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://moz.com/help/moz-procedures/crawlers/rogerbot - JSON: https://www.pathwren.workers.dev/crawler/rogerbot.json Moz's Campaign crawler, which audits a site its own owner registered. Moz states there is no IP range for it — identification is by user-agent only. **Blocking it costs you:** Moz Pro site audits of your domain stop. If the domain is yours and you use Moz, blocking this breaks your own reports. ``` User-agent: rogerbot Disallow: / ``` ### SemrushBot - Operator: Semrush - robots.txt token: `SemrushBot` - User-agent contains: `SemrushBot` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot.json Semrush's backlink and keyword crawler. It is not an AI crawler, but it is usually in the top three by volume on any site, and it is the cheapest block on this list. **Blocking it costs you:** Your competitors' Semrush reports get thinner, and so do yours. No user-facing effect. ``` User-agent: SemrushBot Disallow: / ``` ### SemrushBot-BA - Operator: Semrush - robots.txt token: `SemrushBot-BA` - User-agent contains: `SemrushBot-BA` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-ba.json The Backlink Audit crawler. It re-checks links pointing at a customer's site, which means it lands on the sites doing the linking. **Blocking it costs you:** None to you. It costs the site being audited a little accuracy in their backlink report. ``` User-agent: SemrushBot-BA Disallow: / ``` ### SemrushBot-ESI - Operator: Semrush - robots.txt token: `SemrushBot-ESI` - User-agent contains: `SemrushBot-ESI` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-esi.json The crawler for Semrush Enterprise Site Intelligence, the enterprise tier's own site analysis. **Blocking it costs you:** Enterprise customers lose analysis of your domain. Nothing user-facing. ``` User-agent: SemrushBot-ESI Disallow: / ``` ### SemrushBot-FT - Operator: Semrush - robots.txt token: `SemrushBot-FT` - User-agent contains: `SemrushBot-FT` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-ft.json Fetches full text for the Plagiarism Checker and similar text-comparison tools. **Blocking it costs you:** Your text stops being compared against other people's submissions — which also means copies of your text are less likely to be caught. ``` User-agent: SemrushBot-FT Disallow: / ``` ### SemrushBot-SI - Operator: Semrush - robots.txt token: `SemrushBot-SI` - User-agent contains: `SemrushBot-SI` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-si.json Fetches pages for the On Page SEO Checker and similar advisory tools. **Blocking it costs you:** Nothing user-facing. Semrush customers lose on-page suggestions for pages on your domain. ``` User-agent: SemrushBot-SI Disallow: / ``` ### SemrushBot-SWA - Operator: Semrush - robots.txt token: `SemrushBot-SWA` - User-agent contains: `SemrushBot-SWA` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/semrushbot-swa.json Checks whether a URL is reachable, for the SEO Writing Assistant. One request per URL a writer references, not a crawl. **Blocking it costs you:** Writers using Semrush's assistant see your links reported as unreachable. ``` User-agent: SemrushBot-SWA Disallow: / ``` ### SEOkicks - Operator: SEOkicks - robots.txt token: `SEOkicks` - User-agent contains: `SEOkicks` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.seokicks.de/robot.html - JSON: https://www.pathwren.workers.dev/crawler/seokicks.json A German backlink index. Its documentation names SEOkicks as the user-agent to use in robots.txt. **Blocking it costs you:** You leave a regional backlink index. Nothing else changes. ``` User-agent: SEOkicks Disallow: / ``` ### serpstatbot - Operator: Serpstat - robots.txt token: `serpstatbot` - User-agent contains: `serpstatbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://serpstatbot.com/ - JSON: https://www.pathwren.workers.dev/crawler/serpstatbot.json Serpstat's backlink crawler. It documents support for Crawl-delay up to 20 seconds, including a delay set on the * group. **Blocking it costs you:** You leave Serpstat's link index. Try Crawl-delay first — they honour it, and a slow crawler is cheaper to keep than to fight. ``` User-agent: serpstatbot Disallow: / ``` ### SiteAuditBot - Operator: Semrush - robots.txt token: `SiteAuditBot` - User-agent contains: `SiteAuditBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/siteauditbot.json Semrush's Site Audit crawler: it walks a site a customer owns and reports technical SEO problems. Semrush names it as the token to block for that product. **Blocking it costs you:** Site Audit reports on your own domain stop working. If a customer is auditing your site with your permission, blocking this breaks their tooling and nothing of yours. ``` User-agent: SiteAuditBot Disallow: / ``` ### SplitSignalBot - Operator: Semrush - robots.txt token: `SplitSignalBot` - User-agent contains: `SplitSignalBot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.semrush.com/bot/ - JSON: https://www.pathwren.workers.dev/crawler/splitsignalbot.json Runs SEO A/B tests on a customer's own site with the SplitSignal tool. **Blocking it costs you:** A site owner's own A/B testing stops. Only relevant on domains whose owner uses the product. ``` User-agent: SplitSignalBot Disallow: / ``` ## Archivers Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI one. ### archive.org_bot - Operator: Internet Archive - robots.txt token: `archive.org_bot` - User-agent contains: `archive.org_bot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://archive.org/details/archive.org_bot - JSON: https://www.pathwren.workers.dev/crawler/archive-org-bot.json The Wayback Machine's crawler. Preservation rather than AI, but it lands in the same 'is this bot welcome' decision and its output is a public corpus. **Blocking it costs you:** Your site stops being preserved. When it dies, it is gone. Consider this one separately from the AI question. ``` User-agent: archive.org_bot Disallow: / ``` ### ia_archiver - Operator: Internet Archive - robots.txt token: `ia_archiver` - User-agent contains: `ia_archiver` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://archive.org/details/archive.org_bot - JSON: https://www.pathwren.workers.dev/crawler/ia-archiver.json The legacy Alexa/Internet Archive token, still present in most robots.txt files and still occasionally honoured. **Blocking it costs you:** Negligible today; retain for tidiness. ``` User-agent: ia_archiver Disallow: / ``` ## Link preview fetchers Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident. ### facebookexternalhit - Operator: Meta - robots.txt token: `facebookexternalhit` - User-agent contains: `facebookexternalhit` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - JSON: https://www.pathwren.workers.dev/crawler/facebookexternalhit.json The link unfurler: it reads your Open Graph tags when somebody shares your URL on a Meta property. **Blocking it costs you:** Severe and usually accidental. Your links share as bare grey boxes with no title, image or description across Facebook, Instagram, Messenger and WhatsApp. Almost nobody means to block this. ``` User-agent: facebookexternalhit Disallow: / ``` ### GoogleMessages - Operator: Google - robots.txt token: `GoogleMessages` - User-agent contains: `GoogleMessages` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/googlemessages.json Generates the link preview when somebody sends one of your URLs in Google Messages. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** Your links appear as bare URLs with no title or image in Google Messages chats. A preview fetcher is almost never the one you meant to block. ``` User-agent: GoogleMessages Disallow: / ``` ### Slackbot - Operator: Slack - robots.txt token: `Slackbot` - User-agent contains: `Slackbot` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://api.slack.com/robots - JSON: https://www.pathwren.workers.dev/crawler/slackbot.json The other half of Slack's pair: the agent that reads robots.txt and handles Slack's non-unfurl fetches. Slack documents both strings on one page. **Blocking it costs you:** Slack stops being able to read your robots.txt, which is a strange thing to want. Block the link expander instead if that is the goal. ``` User-agent: Slackbot Disallow: / ``` ### Slackbot-LinkExpanding - Operator: Slack - robots.txt token: `Slackbot-LinkExpanding` - User-agent contains: `Slackbot-LinkExpanding` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://api.slack.com/robots - JSON: https://www.pathwren.workers.dev/crawler/slackbot-linkexpanding.json Fetches a page to build the unfurl card when somebody pastes your link into Slack. One paste, one fetch. **Blocking it costs you:** Your links appear in Slack as bare URLs. Inside working teams that quietly costs you clicks, and nothing is gained. ``` User-agent: Slackbot-LinkExpanding Disallow: / ``` ## Tools and frameworks Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question. ### AdsBot-Google - Operator: Google - robots.txt token: `AdsBot-Google` - User-agent contains: `AdsBot-Google` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers - JSON: https://www.pathwren.workers.dev/crawler/adsbot-google.json Checks the quality of desktop landing pages for Google Ads. Google documents that it ignores the robots.txt * group with the ad publisher's permission, and obeys a group named for its own token. **Blocking it costs you:** Google Ads cannot score your landing pages, which lowers Ad Rank on the ads pointing at them. If you do not buy ads, blocking it costs nothing but bandwidth savings. ``` User-agent: AdsBot-Google Disallow: / ``` ### AdsBot-Google-Mobile - Operator: Google - robots.txt token: `AdsBot-Google-Mobile` - User-agent contains: `AdsBot-Google-Mobile` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers - JSON: https://www.pathwren.workers.dev/crawler/adsbot-google-mobile.json The mobile-web landing page checker for Google Ads. Same rules as AdsBot-Google: the * group does not apply to it, its own token does. **Blocking it costs you:** Mobile ad landing pages go unscored and the ads pointing at them rank worse. No effect on organic search. ``` User-agent: AdsBot-Google-Mobile Disallow: / ``` ### AdsBot-Google-Mobile-Apps - Operator: Google - robots.txt token: `AdsBot-Google-Mobile-Apps` - User-agent contains: `AdsBot-Google-Mobile-Apps` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers - JSON: https://www.pathwren.workers.dev/crawler/adsbot-google-mobile-apps.json Checks Android app landing pages for Google Ads. It obeys a group named for its own token and, per Google, follows the AdsBot-Google rules otherwise. **Blocking it costs you:** App-install ad landing pages go unscored. Nothing organic changes. ``` User-agent: AdsBot-Google-Mobile-Apps Disallow: / ``` ### APIs-Google - Operator: Google - robots.txt token: `APIs-Google` - User-agent contains: `APIs-Google` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers - JSON: https://www.pathwren.workers.dev/crawler/apis-google.json Delivers push notifications for Google APIs to a webhook you registered. It is a special-case crawler: it ignores the robots.txt * group, because the fetch is a delivery to an address you asked it to deliver to. **Blocking it costs you:** Google API push notifications stop arriving at your endpoint. This only affects services you set up yourself; there is no search or AI consequence. ``` User-agent: APIs-Google Disallow: / ``` ### Crawl4AI - Operator: Crawl4AI project - robots.txt token: `Crawl4AI` - User-agent contains: `Crawl4AI` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://github.com/unclecode/crawl4ai - JSON: https://www.pathwren.workers.dev/crawler/crawl4ai.json An open-source LLM-oriented crawler and scraper library, run by whoever installs it. Like Scrapy, the default user-agent identifies the software and says nothing about who is behind the request. **Blocking it costs you:** You block a library, not an operator: the rule catches a researcher and a bulk scraper equally, and anyone who edits one config line is not caught at all. ``` User-agent: Crawl4AI Disallow: / ``` ### Crawlspace - Operator: Crawlspace - robots.txt token: `Crawlspace` - User-agent contains: `Crawlspace` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://crawlspace.dev - JSON: https://www.pathwren.workers.dev/crawler/crawlspace.json A crawling platform: customers run their own crawls on it to feed agents, RAG pipelines and structured-data workflows. Like Firecrawl, the party behind any given request is the customer, not the platform. **Blocking it costs you:** Whatever any Crawlspace customer was building over your pages stops working. Volume and intent vary per customer, so this is a rate-limit decision more than a consent one. ``` User-agent: Crawlspace Disallow: / ``` ### FeedFetcher-Google - Operator: Google - robots.txt token: `FeedFetcher-Google` - User-agent contains: `FeedFetcher-Google` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/feedfetcher-google.json Crawls RSS and Atom feeds for Google News and WebSub. It is a user-triggered fetcher, and Google documents that those generally ignore robots.txt because a person asked for the fetch. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** Feed-driven Google products stop seeing your updates. A robots.txt rule will not stop it — block by user-agent at the edge if you mean it. ``` User-agent: FeedFetcher-Google Disallow: / ``` ### FirecrawlAgent - Operator: Firecrawl - robots.txt token: `FirecrawlAgent` - User-agent contains: `FirecrawlAgent` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://docs.firecrawl.dev/ - JSON: https://www.pathwren.workers.dev/crawler/firecrawlagent.json A hosted scrape-to-markdown service that LLM applications call to read pages. The requester is whoever is building on it, not Firecrawl itself, so volume and intent vary wildly. **Blocking it costs you:** Applications built on Firecrawl cannot read your pages. This is increasingly how agents fetch the web, so it is a bigger block than its name suggests. ``` User-agent: FirecrawlAgent Disallow: / ``` ### Google-CWS - Operator: Google - robots.txt token: `Google-CWS` - User-agent contains: `Google-CWS` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/google-cws.json The Chrome Web Store fetcher. It requests the URLs a developer put in the metadata of a Chrome extension or theme. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** Chrome Web Store listings that point at your pages cannot fetch them. Relevant only if you publish extensions. ``` User-agent: Google-CWS Disallow: / ``` ### Google-InspectionTool - Operator: Google - robots.txt token: `Google-InspectionTool` - User-agent contains: `Google-InspectionTool` - robots.txt: obeys robots.txt (documented) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-inspectiontool.json The fetcher behind Search Console's URL Inspection and the Rich Results Test. It runs when a site owner clicks a button. **Blocking it costs you:** Your own Search Console live tests stop working. Blocking this only hurts you. ``` User-agent: Google-InspectionTool Disallow: / ``` ### Google-Safety - Operator: Google - robots.txt token: `Google-Safety` - User-agent contains: `Google-Safety` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: no published verification method - Docs: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers - JSON: https://www.pathwren.workers.dev/crawler/google-safety.json Google's abuse-investigation fetcher: malware review, phishing reports and similar. Google documents that it ignores robots.txt entirely, and a robots.txt rule for it does nothing. **Blocking it costs you:** Nothing you can control. The rule is ignored by design; listing the token is documentation, not enforcement. ``` User-agent: Google-Safety Disallow: / ``` ### Google-Site-Verification - Operator: Google - robots.txt token: `Google-Site-Verification` - User-agent contains: `Google-Site-Verification` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/google-site-verification.json Fetches the token file or meta tag that proves you own a site, when you click verify in Search Console. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** Your own Search Console verification fails. Blocking this only ever hurts the person doing the blocking. ``` User-agent: Google-Site-Verification Disallow: / ``` ### GoogleProducer - Operator: Google - robots.txt token: `GoogleProducer` - User-agent contains: `GoogleProducer` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, 529 IPv4 / 529 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers - JSON: https://www.pathwren.workers.dev/crawler/googleproducer.json Google Publisher Center: fetches the feeds a publisher explicitly supplied for Google News landing pages. Google publishes fetcher addresses in two files — user-triggered-fetchers.json and user-triggered-fetchers-google.json — and does not say per fetcher which one applies, so verification means checking both; this index mirrors both. **Blocking it costs you:** Your own Google News landing pages stop updating. Only publishers who configured Publisher Center are affected. ``` User-agent: GoogleProducer Disallow: / ``` ### Lightpanda - Operator: Lightpanda - robots.txt token: `Lightpanda` - User-agent contains: `Lightpanda` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://lightpanda.io/ - JSON: https://www.pathwren.workers.dev/crawler/lightpanda.json A purpose-built headless browser for AI and automation — a runtime, not an operator. Whether robots.txt is honoured is left to whoever runs it, which is what its maintainers say themselves. **Blocking it costs you:** You block a browser, not a company: the same rule stops a scraper and a legitimate automation a customer of yours is running. ``` User-agent: Lightpanda Disallow: / ``` ### Mediapartners-Google - Operator: Google - robots.txt token: `Mediapartners-Google` - User-agent contains: `Mediapartners-Google` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: published IP ranges (https://developers.google.com/static/search/apis/ipranges/special-crawlers.json, 136 IPv4 / 136 IPv6) - Docs: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers - JSON: https://www.pathwren.workers.dev/crawler/mediapartners-google.json The AdSense crawler. It reads a page so AdSense can choose relevant ads for it, and it is a special-case crawler that ignores the robots.txt * group. **Blocking it costs you:** Pages it cannot read get generic, lower-value AdSense ads or none at all. This is the one block on this list that costs you money directly if you run AdSense. ``` User-agent: Mediapartners-Google Disallow: / ``` ### Scrapy - Operator: Scrapy project - robots.txt token: `Scrapy` - User-agent contains: `Scrapy` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://scrapy.org/ - JSON: https://www.pathwren.workers.dev/crawler/scrapy.json Not an operator: the default user-agent of the most common Python crawling framework. Anyone can be behind it. Modern Scrapy obeys robots.txt by default, which is why the default UA is still worth a rule. **Blocking it costs you:** You block a very large tail of unattributed one-off crawlers, and also every well-behaved researcher who did not change the default. ``` User-agent: Scrapy Disallow: / ``` ### Screaming Frog SEO Spider - Operator: Screaming Frog - robots.txt token: `Screaming Frog SEO Spider` - User-agent contains: `Screaming Frog SEO Spider` - robots.txt: obeys robots.txt (documented) - Verify by: no published verification method - Docs: https://www.screamingfrog.co.uk/seo-spider/user-agent/ - JSON: https://www.pathwren.workers.dev/crawler/screaming-frog-seo-spider.json Not an operator: desktop crawling software that anybody can point at any site. The default user-agent identifies the tool, not who is running it, and the operator of the moment is whoever pressed start. **Blocking it costs you:** You block a consultant auditing your own site as often as you block a stranger. Treat it as a rate-limit question, not a consent one — and note that the user-agent is configurable, so a block is advisory. ``` User-agent: Screaming Frog SEO Spider Disallow: / ``` ### wpbot - Operator: QuantumCloud - robots.txt token: `wpbot` - User-agent contains: `wpbot` - robots.txt: operator publishes no robots.txt statement - Verify by: no published verification method - Docs: https://www.quantumcloud.com - JSON: https://www.pathwren.workers.dev/crawler/wpbot.json Supports the AI Chatbot for WordPress plugin: it reads pages so the plugin can answer from a site's own content. The operator provides an opt-out through a form rather than through robots.txt. **Blocking it costs you:** A WordPress site running that plugin loses its own content as an answer source. Only relevant where the plugin is installed. ``` User-agent: wpbot Disallow: / ``` ### YandexDirect - Operator: Yandex - robots.txt token: `YandexDirect` - User-agent contains: `YandexDirect` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexdirect.json Reads the content of Yandex Advertising Network partner pages to work out their topic so relevant ads can be matched. Documented as not taking the general robots.txt rules into account. **Blocking it costs you:** Ads on your pages become less relevant and earn less. Relevant only if you monetise with Yandex's network. ``` User-agent: YandexDirect Disallow: / ``` ### YandexMetrika - Operator: Yandex - robots.txt token: `YandexMetrika` - User-agent contains: `YandexMetrika` - robots.txt: not governed by robots.txt (user-initiated, by operator policy) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexmetrika.json Yandex Metrica's own fetcher. Two of its versions — the 2.0 yabs01 availability checker and the 4.0 CSS cache for Webvisor — are documented in Yandex's table as not using robots.txt at all. **Blocking it costs you:** Nothing you can enforce through robots.txt. If you run Metrica, its session replay loses your stylesheets and renders your pages wrong. ``` User-agent: YandexMetrika Disallow: / ``` ### YandexScreenshotBot - Operator: Yandex - robots.txt token: `YandexScreenshotBot` - User-agent contains: `YandexScreenshotBot` - robots.txt: ignores the * group; obeys rules named for its own token - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexscreenshotbot.json Takes a screenshot of a page. Documented as not taking the general robots.txt rules into account. **Blocking it costs you:** Yandex surfaces that show a page thumbnail show nothing for you. ``` User-agent: YandexScreenshotBot Disallow: / ``` ### YandexWebmaster - Operator: Yandex - robots.txt token: `YandexWebmaster` - User-agent contains: `YandexWebmaster` - robots.txt: obeys robots.txt (documented) - Verify by: reverse DNS - Docs: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots - JSON: https://www.pathwren.workers.dev/crawler/yandexwebmaster.json The fetcher behind Yandex Webmaster, the console a site owner uses to inspect their own site. **Blocking it costs you:** Your own Yandex Webmaster checks stop working. Blocking this only hurts you. ``` User-agent: YandexWebmaster Disallow: / ``` --- ## robots.txt policies ### Allow everything, explicitly (`allow-all`, 150 crawlers) Every crawler on this index is named and allowed. Use when you want maximum reach into search and assistants and have nothing to withhold. An empty robots.txt already allows everything, so this file is not about permission — it is about being explicit. Naming each token means a later change is a one-line diff instead of a rewrite, and it documents that the allow was a decision. This is the policy this site itself serves. `curl -s https://www.pathwren.workers.dev/robots/allow-all.txt` ### Block AI training, keep AI search (`block-ai-training`, 27 crawlers) Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers. The distinction most people actually want, and the one that is easy to get wrong: GPTBot trains, OAI-SearchBot indexes for citation. Blocking both loses you the traffic and gains you nothing extra. Google and Apple have no separate crawler at all — Google-Extended and Applebot-Extended are pure control tokens, so they belong in this file while Googlebot and Applebot must not. `curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt` ### Block every AI crawler (`block-all-ai`, 77 crawlers) Training, AI search, user-triggered fetches and corpus builders, all refused. Classic search engines still allowed. The maximal AI opt-out that still leaves you in Google and Bing. Understand the price before deploying it: you will not be cited by any assistant, and when a reader explicitly asks ChatGPT or Claude to open your page, they get an error. Note also that Perplexity-User and Bytespider are listed here but documented as not governed by robots.txt, so this file is a statement of intent for those two, not an enforcement mechanism. `curl -s https://www.pathwren.workers.dev/robots/block-all-ai.txt` ### Block corpus and dataset builders (`block-datasets`, 17 crawlers) Refuse the crawlers whose output is a dataset other people train on: Common Crawl, AI2, Webz.io, Diffbot, ImagesiftBot. These are the highest-leverage blocks per line, because one crawl becomes many downstream training runs. It is also the block with the longest delay before it has any effect, and no effect at all on archives already published. `curl -s https://www.pathwren.workers.dev/robots/block-datasets.txt` ### Allow AI search and user fetches, block the rest (`allow-ai-search-only`, 65 crawlers) Be findable and citable in assistants without contributing to training corpora. The inverse framing of block-ai-training, written as an allowlist so the default for anything new is deny. Fetches a user explicitly asked for stay allowed, because refusing those produces a visible error for a real person who wanted your page. `curl -s https://www.pathwren.workers.dev/robots/allow-ai-search-only.txt` ### Block SEO and backlink crawlers (`block-seo-tools`, 17 crawlers) Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic. The cheapest bandwidth saving available to most sites, and the one nobody regrets. The only cost is that your own dashboards on those tools get thinner. Consider Crawl-delay first: both major operators honour it. `curl -s https://www.pathwren.workers.dev/robots/block-seo-tools.txt` ### Block the crawlers with disputed robots compliance (`block-disputed`, 18 crawlers) The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead. A robots.txt rule is a request. For the operators in this file the request is documented as unreliable or explicitly not applicable, so the honest use of this file is as a record of intent that sits alongside a real block by user-agent or by IP at your CDN. `curl -s https://www.pathwren.workers.dev/robots/block-disputed.txt` ### Maximum AI visibility (`maximum-ai-visibility`, 133 crawlers) Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines. If your content exists to be read by assistants — documentation, reference data, an API — every block costs you and none of them protect anything. Pair this with an llms.txt, a sitemap, and per-item JSON, and the crawlers can actually use what they find. `curl -s https://www.pathwren.workers.dev/robots/maximum-ai-visibility.txt` --- ## Published IP-range endpoints | source | operator | upstream | IPv4 | IPv6 | operator timestamp | |---|---|---|---|---|---| | `openai-gptbot` | OpenAI | https://openai.com/gptbot.json | 21 | 0 | 2025-10-30T11:00:00.000000 | | `openai-searchbot` | OpenAI | https://openai.com/searchbot.json | 35 | 0 | 2026-01-02T11:00:00.000000 | | `openai-chatgpt-user` | OpenAI | https://openai.com/chatgpt-user.json | 204 | 0 | 2026-08-14T20:03:38.055838 | | `google-googlebot` | Google | https://developers.google.com/static/search/apis/ipranges/googlebot.json | 170 | 147 | 2026-09-02T14:45:50.000000 | | `google-special` | Google | https://developers.google.com/static/search/apis/ipranges/special-crawlers.json | 136 | 136 | 2026-09-02T14:45:54.000000 | | `google-user-triggered` | Google | https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json | 529 | 529 | 2026-09-02T14:45:44.000000 | | `google-user-triggered-google` | Google | https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json | 248 | 248 | 2026-09-02T14:45:50.000000 | | `bing-bingbot` | Microsoft | https://www.bing.com/toolbox/bingbot.json | 28 | 0 | 2024-01-03T10:00:00.121331 | | `apple-applebot` | Apple | https://search.developer.apple.com/applebot.json | 33 | 0 | 2026-07-31T10:00:00.000000 | | `duckduckgo-duckduckbot` | DuckDuckGo | https://duckduckgo.com/duckduckbot.json | 486 | 0 | 2026-09-01T12:44:58.000000 | | `perplexity-bot` | Perplexity | https://www.perplexity.ai/perplexitybot.json | 8 | 0 | 2025-02-07T16:56:00.000000 | | `perplexity-user` | Perplexity | https://www.perplexity.ai/perplexity-user.json | 4 | 0 | 2025-10-17T10:17:00.000000 | | `google-user-triggered-agents` | Google | https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json | 3 | 1 | 2026-03-03T10:00:00.000000 | | `commoncrawl-ccbot` | Common Crawl | https://index.commoncrawl.org/ccbot.json | 4 | 1 | 2026-08-11T13:40:00Z | | `ahrefs-crawler` | Ahrefs | https://api.ahrefs.com/v3/public/crawler-ip-ranges | 81 | 0 | — |