curl -s https://www.pathwren.workers.dev/category/tool.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question.
| Crawler | Token | Operator | Cost of blocking |
|---|---|---|---|
| AdsBot-Google | AdsBot-Google | Google Ads cannot score your landing pages, which lowers Ad Rank on the ads pointing at th… | |
| AdsBot-Google-Mobile | AdsBot-Google-Mobile | Mobile ad landing pages go unscored and the ads pointing at them rank worse. No effect on … | |
| AdsBot-Google-Mobile-Apps | AdsBot-Google-Mobile-Apps | App-install ad landing pages go unscored. Nothing organic changes.… | |
| APIs-Google | APIs-Google | Google API push notifications stop arriving at your endpoint. This only affects services y… | |
| Crawl4AI | Crawl4AI | Crawl4AI project | You block a library, not an operator: the rule catches a researcher and a bulk scraper equ… |
| Crawlspace | Crawlspace | Crawlspace | Whatever any Crawlspace customer was building over your pages stops working. Volume and in… |
| FeedFetcher-Google | FeedFetcher-Google | Feed-driven Google products stop seeing your updates. A robots.txt rule will not stop it —… | |
| FirecrawlAgent | FirecrawlAgent | Firecrawl | Applications built on Firecrawl cannot read your pages. This is increasingly how agents fe… |
| Google-CWS | Google-CWS | Chrome Web Store listings that point at your pages cannot fetch them. Relevant only if you… | |
| Google-InspectionTool | Google-InspectionTool | Your own Search Console live tests stop working. Blocking this only hurts you.… | |
| Google-Safety | Google-Safety | Nothing you can control. The rule is ignored by design; listing the token is documentation… | |
| Google-Site-Verification | Google-Site-Verification | Your own Search Console verification fails. Blocking this only ever hurts the person doing… | |
| GoogleProducer | GoogleProducer | Your own Google News landing pages stop updating. Only publishers who configured Publisher… | |
| Lightpanda | Lightpanda | Lightpanda | You block a browser, not a company: the same rule stops a scraper and a legitimate automat… |
| Mediapartners-Google | Mediapartners-Google | Pages it cannot read get generic, lower-value AdSense ads or none at all. This is the one … | |
| Scrapy | Scrapy | Scrapy project | You block a very large tail of unattributed one-off crawlers, and also every well-behaved … |
| Screaming Frog SEO Spider | Screaming Frog SEO Spider | Screaming Frog | You block a consultant auditing your own site as often as you block a stranger. Treat it a… |
| wpbot | wpbot | QuantumCloud | A WordPress site running that plugin loses its own content as an answer source. Only relev… |
| YandexDirect | YandexDirect | Yandex | Ads on your pages become less relevant and earn less. Relevant only if you monetise with Y… |
| YandexMetrika | YandexMetrika | Yandex | Nothing you can enforce through robots.txt. If you run Metrica, its session replay loses y… |
| YandexScreenshotBot | YandexScreenshotBot | Yandex | Yandex surfaces that show a page thumbnail show nothing for you.… |
| YandexWebmaster | YandexWebmaster | Yandex | Your own Yandex Webmaster checks stop working. Blocking this only hurts you.… |