--- title: "Tools and frameworks (22) — AI Crawler Index" description: "Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question." canonical: "https://www.pathwren.workers.dev/category/tool.html" url: "https://www.pathwren.workers.dev/category/tool.md" format: "markdown" source: "the bytes of /category/tool.html, in the build that wrote the page" generator: "surfaces/ai-crawler-index/build.py" generated: "2026-09-03T06:29:54+00:00" license: "CC0-1.0" --- # Tools and frameworks > Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question. Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question. | Crawler | Token | Operator | Cost of blocking | | --- | --- | --- | --- | | [AdsBot-Google](https://www.pathwren.workers.dev/crawler/adsbot-google.html) | `AdsBot-Google` | Google | Google Ads cannot score your landing pages, which lowers Ad Rank on the ads pointing at th… | | [AdsBot-Google-Mobile](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile.html) | `AdsBot-Google-Mobile` | Google | Mobile ad landing pages go unscored and the ads pointing at them rank worse. No effect on … | | [AdsBot-Google-Mobile-Apps](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile-apps.html) | `AdsBot-Google-Mobile-Apps` | Google | App-install ad landing pages go unscored. Nothing organic changes.… | | [APIs-Google](https://www.pathwren.workers.dev/crawler/apis-google.html) | `APIs-Google` | Google | Google API push notifications stop arriving at your endpoint. This only affects services y… | | [Crawl4AI](https://www.pathwren.workers.dev/crawler/crawl4ai.html) | `Crawl4AI` | Crawl4AI project | You block a library, not an operator: the rule catches a researcher and a bulk scraper equ… | | [Crawlspace](https://www.pathwren.workers.dev/crawler/crawlspace.html) | `Crawlspace` | Crawlspace | Whatever any Crawlspace customer was building over your pages stops working. Volume and in… | | [FeedFetcher-Google](https://www.pathwren.workers.dev/crawler/feedfetcher-google.html) | `FeedFetcher-Google` | Google | Feed-driven Google products stop seeing your updates. A robots.txt rule will not stop it —… | | [FirecrawlAgent](https://www.pathwren.workers.dev/crawler/firecrawlagent.html) | `FirecrawlAgent` | Firecrawl | Applications built on Firecrawl cannot read your pages. This is increasingly how agents fe… | | [Google-CWS](https://www.pathwren.workers.dev/crawler/google-cws.html) | `Google-CWS` | Google | Chrome Web Store listings that point at your pages cannot fetch them. Relevant only if you… | | [Google-InspectionTool](https://www.pathwren.workers.dev/crawler/google-inspectiontool.html) | `Google-InspectionTool` | Google | Your own Search Console live tests stop working. Blocking this only hurts you.… | | [Google-Safety](https://www.pathwren.workers.dev/crawler/google-safety.html) | `Google-Safety` | Google | Nothing you can control. The rule is ignored by design; listing the token is documentation… | | [Google-Site-Verification](https://www.pathwren.workers.dev/crawler/google-site-verification.html) | `Google-Site-Verification` | Google | Your own Search Console verification fails. Blocking this only ever hurts the person doing… | | [GoogleProducer](https://www.pathwren.workers.dev/crawler/googleproducer.html) | `GoogleProducer` | Google | Your own Google News landing pages stop updating. Only publishers who configured Publisher… | | [Lightpanda](https://www.pathwren.workers.dev/crawler/lightpanda.html) | `Lightpanda` | Lightpanda | You block a browser, not a company: the same rule stops a scraper and a legitimate automat… | | [Mediapartners-Google](https://www.pathwren.workers.dev/crawler/mediapartners-google.html) | `Mediapartners-Google` | Google | Pages it cannot read get generic, lower-value AdSense ads or none at all. This is the one … | | [Scrapy](https://www.pathwren.workers.dev/crawler/scrapy.html) | `Scrapy` | Scrapy project | You block a very large tail of unattributed one-off crawlers, and also every well-behaved … | | [Screaming Frog SEO Spider](https://www.pathwren.workers.dev/crawler/screaming-frog-seo-spider.html) | `Screaming Frog SEO Spider` | Screaming Frog | You block a consultant auditing your own site as often as you block a stranger. Treat it a… | | [wpbot](https://www.pathwren.workers.dev/crawler/wpbot.html) | `wpbot` | QuantumCloud | A WordPress site running that plugin loses its own content as an answer source. Only relev… | | [YandexDirect](https://www.pathwren.workers.dev/crawler/yandexdirect.html) | `YandexDirect` | Yandex | Ads on your pages become less relevant and earn less. Relevant only if you monetise with Y… | | [YandexMetrika](https://www.pathwren.workers.dev/crawler/yandexmetrika.html) | `YandexMetrika` | Yandex | Nothing you can enforce through robots.txt. If you run Metrica, its session replay loses y… | | [YandexScreenshotBot](https://www.pathwren.workers.dev/crawler/yandexscreenshotbot.html) | `YandexScreenshotBot` | Yandex | Yandex surfaces that show a page thumbnail show nothing for you.… | | [YandexWebmaster](https://www.pathwren.workers.dev/crawler/yandexwebmaster.html) | `YandexWebmaster` | Yandex | Your own Yandex Webmaster checks stop working. Blocking this only hurts you.… | [json](https://www.pathwren.workers.dev/category/tool.json) ## Sitemap - [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates - [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable - [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file - [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag - [A2A agents](https://www.pathwren.workers.dev/a2a.html) - [About and method](https://www.pathwren.workers.dev/about.html) - [API](https://www.pathwren.workers.dev/api.html) - [Changelog](https://www.pathwren.workers.dev/changelog.html) - [Compliance](https://www.pathwren.workers.dev/compliance) - [Contact](https://www.pathwren.workers.dev/contact) - [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum) - [AI Crawler Index](https://www.pathwren.workers.dev/index.html) - [No model runs here](https://www.pathwren.workers.dev/inference.html) - [Legal](https://www.pathwren.workers.dev/legal) - [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html) - [MCP server](https://www.pathwren.workers.dev/mcp-lint.html) - [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html) - [MCP server](https://www.pathwren.workers.dev/mcp-robots.html) - [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html) - [MCP server](https://www.pathwren.workers.dev/mcp-triage.html) - [MCP server](https://www.pathwren.workers.dev/mcp.html) - [Packages](https://www.pathwren.workers.dev/packages.html) - [Pricing](https://www.pathwren.workers.dev/pricing) - [Privacy](https://www.pathwren.workers.dev/privacy.html) - [API reference](https://www.pathwren.workers.dev/reference) - [Access, keys and sign-up](https://www.pathwren.workers.dev/register) - [Security posture](https://www.pathwren.workers.dev/security.html) - [Upstream status](https://www.pathwren.workers.dev/status.html) - [Terms of use](https://www.pathwren.workers.dev/terms.html) - [Trust](https://www.pathwren.workers.dev/trust) ## Machine copies of this page - [HTML (canonical)](https://www.pathwren.workers.dev/category/tool.html) - [JSON](https://www.pathwren.workers.dev/category/tool.json) - [Markdown](https://www.pathwren.workers.dev/category/tool.md) — this document This document is a markdown rendering of [https://www.pathwren.workers.dev/category/tool.html](https://www.pathwren.workers.dev/category/tool.html), generated from that page's own bytes in the same build. The HTML page is canonical.