---
title: "Search engines (28) — AI Crawler Index"
description: "Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block."
canonical: "https://www.pathwren.workers.dev/category/search.html"
url: "https://www.pathwren.workers.dev/category/search.md"
format: "markdown"
source: "the bytes of /category/search.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# Search engines

> Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.

Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block.

| Crawler | Token | Operator | Cost of blocking |
| --- | --- | --- | --- |
| [Applebot](https://www.pathwren.workers.dev/crawler/applebot.html) | `Applebot` | Apple | You disappear from Siri, Spotlight and Safari search suggestions across Apple's install ba… |
| [Baiduspider](https://www.pathwren.workers.dev/crawler/baiduspider.html) | `Baiduspider` | Baidu | Removal from Baidu Search, which matters only if you want Chinese-language traffic.… |
| [bingbot](https://www.pathwren.workers.dev/crawler/bingbot.html) | `bingbot` | Microsoft | Very high and very wide: Bing, Copilot, DuckDuckGo and several assistants that resell Bing… |
| [DuckDuckBot](https://www.pathwren.workers.dev/crawler/duckduckbot.html) | `DuckDuckBot` | DuckDuckGo | Limited on its own; the real DuckDuckGo lever is bingbot.… |
| [Googlebot](https://www.pathwren.workers.dev/crawler/googlebot.html) | `Googlebot` | Google | Total. You leave Google Search. Never block this to avoid AI use; use Google-Extended inst… |
| [Googlebot-Image](https://www.pathwren.workers.dev/crawler/googlebot-image.html) | `Googlebot-Image` | Google | Your images stop appearing in Google Images.… |
| [Googlebot-News](https://www.pathwren.workers.dev/crawler/googlebot-news.html) | `Googlebot-News` | Google | Removal from Google News, with normal Search unaffected.… |
| [Googlebot-Video](https://www.pathwren.workers.dev/crawler/googlebot-video.html) | `Googlebot-Video` | Google | Your videos leave Google video search. A rule for Googlebot already covers it, so blocking… |
| [Kagibot](https://www.pathwren.workers.dev/crawler/kagibot.html) | `Kagibot` | Kagi | You leave Kagi's index. Kagi's users are paying to search and skew technical; per visitor … |
| [MojeekBot](https://www.pathwren.workers.dev/crawler/mojeekbot.html) | `MojeekBot` | Mojeek | You leave an independent index that other privacy-focused search products draw on. Small t… |
| [PetalBot](https://www.pathwren.workers.dev/crawler/petalbot.html) | `PetalBot` | Huawei | Removal from Petal Search. Frequently blocked for volume rather than for policy.… |
| [Pinterestbot](https://www.pathwren.workers.dev/crawler/pinterestbot.html) | `Pinterestbot` | Pinterest | Pins pointing at your site go stale — wrong prices, dead links — and new content stops bei… |
| [Qwantbot](https://www.pathwren.workers.dev/crawler/qwantbot.html) | `Qwantbot` | Qwant | You leave the index behind Qwant, the French privacy-focused engine, and the products that… |
| [Qwantbot-news](https://www.pathwren.workers.dev/crawler/qwantbot-news.html) | `Qwantbot-news` | Qwant | Your articles stop appearing in Qwant News. A rule for Qwantbot as a substring already cat… |
| [SeznamBot](https://www.pathwren.workers.dev/crawler/seznambot.html) | `SeznamBot` | Seznam | Removal from Seznam. Also removes you from its IndexNow endpoint's usefulness.… |
| [Storebot-Google](https://www.pathwren.workers.dev/crawler/storebot-google.html) | `Storebot-Google` | Google | Product listings may lose shopping-specific enrichment. Irrelevant to non-commerce sites.… |
| [Timpibot](https://www.pathwren.workers.dev/crawler/timpibot.html) | `Timpibot` | Timpi | Absence from a small independent index.… |
| [YandexBlogs](https://www.pathwren.workers.dev/crawler/yandexblogs.html) | `YandexBlogs` | Yandex | Comment threads and blog posts stop being findable through Yandex blog search.… |
| [YandexBot](https://www.pathwren.workers.dev/crawler/yandexbot.html) | `YandexBot` | Yandex | Removal from Yandex Search. Verify with reverse DNS to a yandex.ru, yandex.net or yandex.c… |
| [YandexComBot](https://www.pathwren.workers.dev/crawler/yandexcombot.html) | `YandexComBot` | Yandex | You leave Yandex's non-Russian index. A rule naming this token is the only one that works … |
| [YandexFavicons](https://www.pathwren.workers.dev/crawler/yandexfavicons.html) | `YandexFavicons` | Yandex | Your results in Yandex lose their icon. Cosmetic, and a * rule will not achieve it anyway.… |
| [YandexImages](https://www.pathwren.workers.dev/crawler/yandeximages.html) | `YandexImages` | Yandex | Your images leave Yandex Images, which is a large share of image search in Russian-speakin… |
| [YandexMarket](https://www.pathwren.workers.dev/crawler/yandexmarket.html) | `YandexMarket` | Yandex | Your products stop being listed and priced in Yandex Market. For a retailer in that market… |
| [YandexMedia](https://www.pathwren.workers.dev/crawler/yandexmedia.html) | `YandexMedia` | Yandex | Your multimedia content stops appearing in Yandex's media surfaces.… |
| [YandexMobileBot](https://www.pathwren.workers.dev/crawler/yandexmobilebot.html) | `YandexMobileBot` | Yandex | Yandex loses its mobile-friendliness signal for your pages, which affects how they are ran… |
| [YandexRenderResourcesBot](https://www.pathwren.workers.dev/crawler/yandexrenderresourcesbot.html) | `YandexRenderResourcesBot` | Yandex | Yandex renders your pages without their stylesheets or scripts and ranks what it sees. Thi… |
| [YandexVideo](https://www.pathwren.workers.dev/crawler/yandexvideo.html) | `YandexVideo` | Yandex | Removal from Yandex video search. Note that a second robot, YandexVideoParser, does the sa… |
| [Yeti](https://www.pathwren.workers.dev/crawler/yeti.html) | `Yeti` | Naver | Removal from Naver, which is most of Korean search.… |

[json](https://www.pathwren.workers.dev/category/search.json)

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/category/search.html)
- [JSON](https://www.pathwren.workers.dev/category/search.json)
- [Markdown](https://www.pathwren.workers.dev/category/search.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/category/search.html](https://www.pathwren.workers.dev/category/search.html), generated from that page's own bytes in the same build. The HTML page is canonical.