curl -s https://www.pathwren.workers.dev/category/search.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| Applebot | Applebot | Apple | You disappear from Siri, Spotlight and Safari search suggestions across Apple's install ba… |
| Baiduspider | Baiduspider | Baidu | Removal from Baidu Search, which matters only if you want Chinese-language traffic.… |
| bingbot | bingbot | Microsoft | Very high and very wide: Bing, Copilot, DuckDuckGo and several assistants that resell Bing… |
| DuckDuckBot | DuckDuckBot | DuckDuckGo | Limited on its own; the real DuckDuckGo lever is bingbot.… |
| Googlebot | Googlebot | Total. You leave Google Search. Never block this to avoid AI use; use Google-Extended inst… | |
| Googlebot-Image | Googlebot-Image | Your images stop appearing in Google Images.… | |
| Googlebot-News | Googlebot-News | Removal from Google News, with normal Search unaffected.… | |
| Googlebot-Video | Googlebot-Video | Your videos leave Google video search. A rule for Googlebot already covers it, so blocking… | |
| Kagibot | Kagibot | Kagi | You leave Kagi's index. Kagi's users are paying to search and skew technical; per visitor … |
| MojeekBot | MojeekBot | Mojeek | You leave an independent index that other privacy-focused search products draw on. Small t… |
| PetalBot | PetalBot | Huawei | Removal from Petal Search. Frequently blocked for volume rather than for policy.… |
| Pinterestbot | Pinterestbot | Pins pointing at your site go stale — wrong prices, dead links — and new content stops bei… | |
| Qwantbot | Qwantbot | Qwant | You leave the index behind Qwant, the French privacy-focused engine, and the products that… |
| Qwantbot-news | Qwantbot-news | Qwant | Your articles stop appearing in Qwant News. A rule for Qwantbot as a substring already cat… |
| SeznamBot | SeznamBot | Seznam | Removal from Seznam. Also removes you from its IndexNow endpoint's usefulness.… |
| Storebot-Google | Storebot-Google | Product listings may lose shopping-specific enrichment. Irrelevant to non-commerce sites.… | |
| Timpibot | Timpibot | Timpi | Absence from a small independent index.… |
| YandexBlogs | YandexBlogs | Yandex | Comment threads and blog posts stop being findable through Yandex blog search.… |
| YandexBot | YandexBot | Yandex | Removal from Yandex Search. Verify with reverse DNS to a yandex.ru, yandex.net or yandex.c… |
| YandexComBot | YandexComBot | Yandex | You leave Yandex's non-Russian index. A rule naming this token is the only one that works … |
| YandexFavicons | YandexFavicons | Yandex | Your results in Yandex lose their icon. Cosmetic, and a * rule will not achieve it anyway.… |
| YandexImages | YandexImages | Yandex | Your images leave Yandex Images, which is a large share of image search in Russian-speakin… |
| YandexMarket | YandexMarket | Yandex | Your products stop being listed and priced in Yandex Market. For a retailer in that market… |
| YandexMedia | YandexMedia | Yandex | Your multimedia content stops appearing in Yandex's media surfaces.… |
| YandexMobileBot | YandexMobileBot | Yandex | Yandex loses its mobile-friendliness signal for your pages, which affects how they are ran… |
| YandexRenderResourcesBot | YandexRenderResourcesBot | Yandex | Yandex renders your pages without their stylesheets or scripts and ranks what it sees. Thi… |
| YandexVideo | YandexVideo | Yandex | Removal from Yandex video search. Note that a second robot, YandexVideoParser, does the sa… |
| Yeti | Yeti | Naver | Removal from Naver, which is most of Korean search.… |