--- title: "Google crawlers (26) — AI Crawler Index" description: "Every crawler Google operates, its robots.txt token, and what blocking it costs." canonical: "https://www.pathwren.workers.dev/operator/google.html" url: "https://www.pathwren.workers.dev/operator/google.md" format: "markdown" source: "the bytes of /operator/google.html, in the build that wrote the page" generator: "surfaces/ai-crawler-index/build.py" generated: "2026-09-03T07:55:35+00:00" license: "CC0-1.0" --- # Google > Every crawler Google operates, its robots.txt token, and what blocking it costs. 26 crawler(s) · [json](https://www.pathwren.workers.dev/operator/google.json) | Crawler | Token | Category | robots.txt | | --- | --- | --- | --- | | [AdsBot-Google](https://www.pathwren.workers.dev/crawler/adsbot-google.html) | `AdsBot-Google` | Tools and frameworks | ignores the * group; obeys rules named for its own token | | [AdsBot-Google-Mobile](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile.html) | `AdsBot-Google-Mobile` | Tools and frameworks | ignores the * group; obeys rules named for its own token | | [AdsBot-Google-Mobile-Apps](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile-apps.html) | `AdsBot-Google-Mobile-Apps` | Tools and frameworks | ignores the * group; obeys rules named for its own token | | [APIs-Google](https://www.pathwren.workers.dev/crawler/apis-google.html) | `APIs-Google` | Tools and frameworks | ignores the * group; obeys rules named for its own token | | [FeedFetcher-Google](https://www.pathwren.workers.dev/crawler/feedfetcher-google.html) | `FeedFetcher-Google` | Tools and frameworks | not governed by robots.txt (user-initiated, by operator policy) | | [Google-Agent](https://www.pathwren.workers.dev/crawler/google-agent.html) | `Google-Agent` | User-triggered fetchers | not governed by robots.txt (user-initiated, by operator policy) | | [Google-CloudVertexBot](https://www.pathwren.workers.dev/crawler/google-cloudvertexbot.html) | `Google-CloudVertexBot` | AI search crawlers | obeys robots.txt (documented) | | [Google-CWS](https://www.pathwren.workers.dev/crawler/google-cws.html) | `Google-CWS` | Tools and frameworks | not governed by robots.txt (user-initiated, by operator policy) | | [Google-Extended](https://www.pathwren.workers.dev/crawler/google-extended.html) | `Google-Extended` | AI training crawlers | control token only — no crawler | | [Google-GeminiNotebook](https://www.pathwren.workers.dev/crawler/google-gemininotebook.html) | `Google-GeminiNotebook` | User-triggered fetchers | not governed by robots.txt (user-initiated, by operator policy) | | [Google-InspectionTool](https://www.pathwren.workers.dev/crawler/google-inspectiontool.html) | `Google-InspectionTool` | Tools and frameworks | obeys robots.txt (documented) | | [Google-Pinpoint](https://www.pathwren.workers.dev/crawler/google-pinpoint.html) | `Google-Pinpoint` | User-triggered fetchers | not governed by robots.txt (user-initiated, by operator policy) | | [Google-Read-Aloud](https://www.pathwren.workers.dev/crawler/google-read-aloud.html) | `Google-Read-Aloud` | User-triggered fetchers | not governed by robots.txt (user-initiated, by operator policy) | | [Google-Safety](https://www.pathwren.workers.dev/crawler/google-safety.html) | `Google-Safety` | Tools and frameworks | not governed by robots.txt (user-initiated, by operator policy) | | [Google-Site-Verification](https://www.pathwren.workers.dev/crawler/google-site-verification.html) | `Google-Site-Verification` | Tools and frameworks | not governed by robots.txt (user-initiated, by operator policy) | | [Googlebot](https://www.pathwren.workers.dev/crawler/googlebot.html) | `Googlebot` | Search engines | obeys robots.txt (documented) | | [Googlebot-Image](https://www.pathwren.workers.dev/crawler/googlebot-image.html) | `Googlebot-Image` | Search engines | obeys robots.txt (documented) | | [Googlebot-News](https://www.pathwren.workers.dev/crawler/googlebot-news.html) | `Googlebot-News` | Search engines | obeys robots.txt (documented) | | [Googlebot-Video](https://www.pathwren.workers.dev/crawler/googlebot-video.html) | `Googlebot-Video` | Search engines | obeys robots.txt (documented) | | [GoogleMessages](https://www.pathwren.workers.dev/crawler/googlemessages.html) | `GoogleMessages` | Link preview fetchers | not governed by robots.txt (user-initiated, by operator policy) | | [GoogleOther](https://www.pathwren.workers.dev/crawler/googleother.html) | `GoogleOther` | AI training crawlers | obeys robots.txt (documented) | | [GoogleOther-Image](https://www.pathwren.workers.dev/crawler/googleother-image.html) | `GoogleOther-Image` | AI training crawlers | obeys robots.txt (documented) | | [GoogleOther-Video](https://www.pathwren.workers.dev/crawler/googleother-video.html) | `GoogleOther-Video` | AI training crawlers | obeys robots.txt (documented) | | [GoogleProducer](https://www.pathwren.workers.dev/crawler/googleproducer.html) | `GoogleProducer` | Tools and frameworks | not governed by robots.txt (user-initiated, by operator policy) | | [Mediapartners-Google](https://www.pathwren.workers.dev/crawler/mediapartners-google.html) | `Mediapartners-Google` | Tools and frameworks | ignores the * group; obeys rules named for its own token | | [Storebot-Google](https://www.pathwren.workers.dev/crawler/storebot-google.html) | `Storebot-Google` | Search engines | obeys robots.txt (documented) | ## Published IP ranges | Endpoint | IPv4 | IPv6 | Operator timestamp | | --- | --- | --- | --- | | [Googlebot](https://www.pathwren.workers.dev/ip-ranges/google-googlebot.html) | 170 | 147 | 2026-09-02T14:45:50.000000 | | [Google special-purpose crawlers](https://www.pathwren.workers.dev/ip-ranges/google-special.html) | 136 | 136 | 2026-09-02T14:45:54.000000 | | [Google user-triggered fetchers](https://www.pathwren.workers.dev/ip-ranges/google-user-triggered.html) | 529 | 529 | 2026-09-02T14:45:44.000000 | | [Google user-triggered fetchers (Google-owned ranges)](https://www.pathwren.workers.dev/ip-ranges/google-user-triggered-google.html) | 248 | 248 | 2026-09-02T14:45:50.000000 | | [Google-Agent (agents on Google infrastructure)](https://www.pathwren.workers.dev/ip-ranges/google-user-triggered-agents.html) | 3 | 1 | 2026-03-03T10:00:00.000000 | Documentation: [https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) ## Sitemap - [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates - [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable - [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file - [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag - [A2A agents](https://www.pathwren.workers.dev/a2a.html) - [About and method](https://www.pathwren.workers.dev/about.html) - [API](https://www.pathwren.workers.dev/api.html) - [Changelog](https://www.pathwren.workers.dev/changelog.html) - [Compliance](https://www.pathwren.workers.dev/compliance) - [Contact](https://www.pathwren.workers.dev/contact) - [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum) - [AI Crawler Index](https://www.pathwren.workers.dev/index.html) - [No model runs here](https://www.pathwren.workers.dev/inference.html) - [Legal](https://www.pathwren.workers.dev/legal) - [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html) - [MCP server](https://www.pathwren.workers.dev/mcp-lint.html) - [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html) - [MCP server](https://www.pathwren.workers.dev/mcp-robots.html) - [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html) - [MCP server](https://www.pathwren.workers.dev/mcp-triage.html) - [MCP server](https://www.pathwren.workers.dev/mcp.html) - [Packages](https://www.pathwren.workers.dev/packages.html) - [Pricing](https://www.pathwren.workers.dev/pricing) - [Privacy](https://www.pathwren.workers.dev/privacy.html) - [API reference](https://www.pathwren.workers.dev/reference) - [Access, keys and sign-up](https://www.pathwren.workers.dev/register) - [Security posture](https://www.pathwren.workers.dev/security.html) - [Upstream status](https://www.pathwren.workers.dev/status.html) - [Terms of use](https://www.pathwren.workers.dev/terms.html) - [Trust](https://www.pathwren.workers.dev/trust) ## Machine copies of this page - [HTML (canonical)](https://www.pathwren.workers.dev/operator/google.html) - [JSON](https://www.pathwren.workers.dev/operator/google.json) - [Markdown](https://www.pathwren.workers.dev/operator/google.md) — this document This document is a markdown rendering of [https://www.pathwren.workers.dev/operator/google.html](https://www.pathwren.workers.dev/operator/google.html), generated from that page's own bytes in the same build. The HTML page is canonical.