curl -s https://www.pathwren.workers.dev/category/ai-search.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| AIWebIndex | AIWebIndex | Lyrenth | Agents reading through this index stop seeing you — including the attribution and link bac… |
| Amazonbot | Amazonbot | Amazon | Alexa and Amazon's assistants stop answering from your pages. Verify with reverse DNS to c… |
| Andibot | Andibot | Andi | You disappear from another assistant's answers. Andi publishes no robots.txt statement.… |
| Anomura | Anomura | Direqt | If you are the publisher, this breaks the assistant you put on your own pages. If you are … |
| atlassian-bot | atlassian-bot | Atlassian | Rovo cannot answer from your public documentation. If your customers live inside Atlassian… |
| bedrockbot | bedrockbot | Amazon | Companies building retrieval applications on Bedrock cannot include your pages. This is a … |
| Claude-SearchBot | Claude-SearchBot | Anthropic | You stop appearing in Claude's search results and citations.… |
| Claude-Web | Claude-Web | Anthropic | None in practice. Retain the rule; expect no traffic.… |
| Cloudflare-AutoRAG | Cloudflare-AutoRAG | Cloudflare | Applications built on Cloudflare AI Search cannot retrieve your pages. If you are the one … |
| DuckAssistBot | DuckAssistBot | DuckDuckGo | No DuckAssist answers or citations from your site. Ordinary DuckDuckGo results are unaffec… |
| ExaSearchBot | ExaSearchBot | Exa | Agents built on Exa's API stop finding you. Exa publishes no statement about robots.txt co… |
| Google-CloudVertexBot | Google-CloudVertexBot | Third parties can no longer build Vertex AI agents that read your site. Irrelevant to Goog… | |
| KlaviyoAIBot | KlaviyoAIBot | Klaviyo | If the connected domain is yours, blocking this breaks the agent you configured. If it is … |
| Meta-WebIndexer | Meta-WebIndexer | Meta | You leave the index Meta AI answers from across Facebook, Instagram and WhatsApp — the lar… |
| OAI-SearchBot | OAI-SearchBot | OpenAI | High. Blocking this removes you from ChatGPT search results and from the source links Chat… |
| PerplexityBot | PerplexityBot | Perplexity | You stop being indexed and cited by Perplexity, and lose the referral clicks its citations… |
| PhindBot | PhindBot | Phind | You stop being cited in answers to technical questions — which, for documentation and refe… |
| QualifiedBot | QualifiedBot | Qualified | A chatbot on a site that licensed the product loses context. If that site is yours, this b… |
| ShapBot | ShapBot | Parallel | Agents using Parallel's research API lose you as a source. Parallel documents robots.txt c… |
| TerraCotta | TerraCotta | Ceramic AI | You are absent from another agent-facing retrieval index. Ceramic documents that it obeys … |
| YouBot | YouBot | You.com | Removal from You.com's index and from answers built on its API.… |