curl -s https://www.pathwren.workers.dev/category/dataset.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| AI2Bot | AI2Bot | Allen Institute for AI | Excluded from open research datasets. Worth a deliberate decision: this is the category wh… |
| Ai2Bot-Dolma | Ai2Bot-Dolma | Allen Institute for AI | Same as AI2Bot: exclusion from an open, published training corpus.… |
| aiHitBot | aiHitBot | aiHit | Your company record in a B2B dataset goes stale. Documented as respecting robots.txt, so t… |
| AwarioRssBot | AwarioRssBot | Awario | Your RSS updates stop reaching Awario's monitoring. Block both tokens or neither.… |
| AwarioSmartBot | AwarioSmartBot | Awario | Mentions of brands on your pages stop being surfaced to the people monitoring them — inclu… |
| CCBot | CCBot | Common Crawl | Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but … |
| Diffbot | Diffbot | Diffbot | Your facts stop entering a widely-licensed knowledge graph. Whether that is a loss depends… |
| EchoboxBot | EchoboxBot | Echobox | Publishers using Echobox get worse scheduling decisions about your articles. No compliance… |
| ImagesiftBot | ImagesiftBot | Hive AI | Your images stop entering an image dataset and reverse-image index.… |
| img2dataset | img2dataset | LAION / img2dataset | Your images are skipped when someone materialises an image-text dataset that references th… |
| LAIONDownloader | LAIONDownloader | LAION / img2dataset | Your media is skipped when an open research dataset is built from URL lists. Once a datase… |
| omgili | omgili | Webz.io | Same as omgilibot.… |
| omgilibot | omgilibot | Webz.io | Exclusion from a commercial dataset resold to third parties.… |
| Panscient | panscient.com | Panscient | Your company pages stop feeding a business-data product. No effect on search or assistants… |
| Thinkbot | Thinkbot | Thinkbot | Exclusion from a market-research dataset. Expect to enforce this at the edge rather than i… |
| VelenPublicWebCrawler | VelenPublicWebCrawler | Hunter (Velen) | Your company pages stop feeding a B2B contact and company dataset. The crawl rate it docum… |
| YaK | YaK | Meltwater | Your content stops appearing in Meltwater's media monitoring — which is how PR teams find … |