curl -s https://www.pathwren.workers.dev/crawler/cloudflare-autorag.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Cloudflare · AI search crawlers · json
User-agent: Cloudflare-AutoRAG Disallow: /
| robots.txt token | Cloudflare-AutoRAG |
| User-agent contains | Cloudflare-AutoRAG |
| Operator | Cloudflare |
| Category | AI search crawlers |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
The crawler behind Cloudflare's AI Search / AutoRAG, which indexes a website into a retrieval index for an application. Cloudflare's own documentation warns that a bot-blocking rule on your zone will also stop this crawler and tells you to allow-list it.
Applications built on Cloudflare AI Search cannot retrieve your pages. If you are the one building the index over your own site, blocking it breaks your own product.
Cloudflare-AutoRAG
User-agent: Cloudflare-AutoRAG Allow: /
Operator documentation: https://developers.cloudflare.com/ai-search/
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-all-ai · allow-ai-search-only · maximum-ai-visibility