curl -s https://www.pathwren.workers.dev/operator/index.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
| Operator | Crawlers | Categories |
|---|---|---|
| Ahrefs | 2 | SEO and backlink crawlers |
| aiHit | 1 | Corpus and dataset builders |
| Allen Institute for AI | 2 | Corpus and dataset builders |
| Amazon | 2 | AI search crawlers |
| Andi | 1 | AI search crawlers |
| Anthropic | 5 | AI search crawlers, AI training crawlers, User-triggered fetchers |
| Apple | 2 | AI training crawlers, Search engines |
| Atlassian | 1 | AI search crawlers |
| Awario | 2 | Corpus and dataset builders |
| Babbar | 1 | SEO and backlink crawlers |
| Baidu | 1 | Search engines |
| ByteDance | 2 | AI training crawlers |
| Ceramic AI | 1 | AI search crawlers |
| Cloudflare | 1 | AI search crawlers |
| Cohere | 2 | AI training crawlers, User-triggered fetchers |
| Common Crawl | 1 | Corpus and dataset builders |
| Crawl4AI project | 1 | Tools and frameworks |
| Crawlspace | 1 | Tools and frameworks |
| DataForSEO | 1 | SEO and backlink crawlers |
| Diffbot | 1 | Corpus and dataset builders |
| Direqt | 1 | AI search crawlers |
| DuckDuckGo | 2 | AI search crawlers, Search engines |
| Echobox | 1 | Corpus and dataset builders |
| Exa | 1 | AI search crawlers |
| FactSet | 1 | AI training crawlers |
| Firecrawl | 1 | Tools and frameworks |
| 26 | AI search crawlers, AI training crawlers, Link preview fetchers, Search engines, Tools and frameworks, User-triggered fetchers | |
| Hive AI | 1 | Corpus and dataset builders |
| Huawei | 1 | Search engines |
| Hunter (Velen) | 1 | Corpus and dataset builders |
| Internet Archive | 2 | Archivers |
| ISS Corporate Solutions | 1 | AI training crawlers |
| Kagi | 1 | Search engines |
| Klaviyo | 1 | AI search crawlers |
| LAION / img2dataset | 2 | Corpus and dataset builders |
| Lightpanda | 1 | Tools and frameworks |
| Linguee | 1 | AI training crawlers |
| Lyrenth | 1 | AI search crawlers |
| Majestic | 1 | SEO and backlink crawlers |
| Meltwater | 1 | Corpus and dataset builders |
| Meta | 5 | AI search crawlers, AI training crawlers, Link preview fetchers, User-triggered fetchers |
| Microsoft | 1 | Search engines |
| Mistral AI | 1 | User-triggered fetchers |
| Mojeek | 1 | Search engines |
| Moz | 2 | SEO and backlink crawlers |
| Naver | 1 | Search engines |
| NICT | 1 | AI training crawlers |
| OpenAI | 4 | AI search crawlers, AI training crawlers, User-triggered fetchers |
| Panscient | 1 | Corpus and dataset builders |
| Parallel | 1 | AI search crawlers |
| Perplexity | 2 | AI search crawlers, User-triggered fetchers |
| Phind | 1 | AI search crawlers |
| 1 | Search engines | |
| Poseidon Research | 1 | AI training crawlers |
| Qualified | 1 | AI search crawlers |
| QuantumCloud | 1 | Tools and frameworks |
| QuillBot | 1 | AI training crawlers |
| Qwant | 2 | Search engines |
| Reflection AI | 1 | AI training crawlers |
| ROIS-DS | 1 | AI training crawlers |
| SB Intuitions | 1 | AI training crawlers |
| Scrapy project | 1 | Tools and frameworks |
| Screaming Frog | 1 | Tools and frameworks |
| Semrush | 9 | AI training crawlers, SEO and backlink crawlers |
| SEOkicks | 1 | SEO and backlink crawlers |
| Serpstat | 1 | SEO and backlink crawlers |
| Seznam | 1 | Search engines |
| Sidetrade | 1 | AI training crawlers |
| Slack | 2 | Link preview fetchers |
| Thinkbot | 1 | Corpus and dataset builders |
| Timpi | 1 | Search engines |
| Webz.io | 3 | AI training crawlers, Corpus and dataset builders |
| Yandex | 17 | AI training crawlers, Search engines, Tools and frameworks, User-triggered fetchers |
| You.com | 1 | AI search crawlers |