---
title: "robots.txt: Maximum AI visibility — AI Crawler Index"
description: "Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines."
canonical: "https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html"
url: "https://www.pathwren.workers.dev/policy/maximum-ai-visibility.md"
format: "markdown"
source: "the bytes of /policy/maximum-ai-visibility.html, in the build that wrote the page"
generator: "surfaces/ai-crawler-index/build.py"
generated: "2026-09-03T06:29:54+00:00"
license: "CC0-1.0"
---

# Maximum AI visibility

> Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines.

Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines.

```bash
curl -s https://www.pathwren.workers.dev/robots/maximum-ai-visibility.txt >> robots.txt
```

If your content exists to be read by assistants — documentation, reference data, an API — every block costs you and none of them protect anything. Pair this with an llms.txt, a sitemap, and per-item JSON, and the crawlers can actually use what they find.

## Names 133 crawlers

[AdsBot-Google](https://www.pathwren.workers.dev/crawler/adsbot-google.html) · [AdsBot-Google-Mobile](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile.html) · [AdsBot-Google-Mobile-Apps](https://www.pathwren.workers.dev/crawler/adsbot-google-mobile-apps.html) · [AI2Bot](https://www.pathwren.workers.dev/crawler/ai2bot.html) · [Ai2Bot-Dolma](https://www.pathwren.workers.dev/crawler/ai2bot-dolma.html) · [aiHitBot](https://www.pathwren.workers.dev/crawler/aihitbot.html) · [AIWebIndex](https://www.pathwren.workers.dev/crawler/aiwebindex.html) · [Amazonbot](https://www.pathwren.workers.dev/crawler/amazonbot.html) · [Andibot](https://www.pathwren.workers.dev/crawler/andibot.html) · [Anomura](https://www.pathwren.workers.dev/crawler/anomura.html) · [anthropic-ai](https://www.pathwren.workers.dev/crawler/anthropic-ai.html) · [APIs-Google](https://www.pathwren.workers.dev/crawler/apis-google.html) · [Applebot](https://www.pathwren.workers.dev/crawler/applebot.html) · [Applebot-Extended](https://www.pathwren.workers.dev/crawler/applebot-extended.html) · [archive.org_bot](https://www.pathwren.workers.dev/crawler/archive-org-bot.html) · [atlassian-bot](https://www.pathwren.workers.dev/crawler/atlassian-bot.html) · [AwarioRssBot](https://www.pathwren.workers.dev/crawler/awariorssbot.html) · [AwarioSmartBot](https://www.pathwren.workers.dev/crawler/awariosmartbot.html) · [Baiduspider](https://www.pathwren.workers.dev/crawler/baiduspider.html) · [bedrockbot](https://www.pathwren.workers.dev/crawler/bedrockbot.html) · [bingbot](https://www.pathwren.workers.dev/crawler/bingbot.html) · [Bytespider](https://www.pathwren.workers.dev/crawler/bytespider.html) · [CCBot](https://www.pathwren.workers.dev/crawler/ccbot.html) · [ChatGPT Agent](https://www.pathwren.workers.dev/crawler/chatgpt-agent.html) · [ChatGPT-User](https://www.pathwren.workers.dev/crawler/chatgpt-user.html) · [Claude-SearchBot](https://www.pathwren.workers.dev/crawler/claude-searchbot.html) · [Claude-User](https://www.pathwren.workers.dev/crawler/claude-user.html) · [Claude-Web](https://www.pathwren.workers.dev/crawler/claude-web.html) · [ClaudeBot](https://www.pathwren.workers.dev/crawler/claudebot.html) · [Cloudflare-AutoRAG](https://www.pathwren.workers.dev/crawler/cloudflare-autorag.html) · [cohere-ai](https://www.pathwren.workers.dev/crawler/cohere-ai.html) · [cohere-training-data-crawler](https://www.pathwren.workers.dev/crawler/cohere-training-data-crawler.html) · [Cotoyogi](https://www.pathwren.workers.dev/crawler/cotoyogi.html) · [Crawl4AI](https://www.pathwren.workers.dev/crawler/crawl4ai.html) · [Crawlspace](https://www.pathwren.workers.dev/crawler/crawlspace.html) · [Diffbot](https://www.pathwren.workers.dev/crawler/diffbot.html) · [DuckAssistBot](https://www.pathwren.workers.dev/crawler/duckassistbot.html) · [DuckDuckBot](https://www.pathwren.workers.dev/crawler/duckduckbot.html) · [EchoboxBot](https://www.pathwren.workers.dev/crawler/echoboxbot.html) · [ExaSearchBot](https://www.pathwren.workers.dev/crawler/exasearchbot.html) · [FacebookBot](https://www.pathwren.workers.dev/crawler/facebookbot.html) · [facebookexternalhit](https://www.pathwren.workers.dev/crawler/facebookexternalhit.html) · [Factset_spyderbot](https://www.pathwren.workers.dev/crawler/factset-spyderbot.html) · [FeedFetcher-Google](https://www.pathwren.workers.dev/crawler/feedfetcher-google.html) · [FirecrawlAgent](https://www.pathwren.workers.dev/crawler/firecrawlagent.html) · [Google-Agent](https://www.pathwren.workers.dev/crawler/google-agent.html) · [Google-CloudVertexBot](https://www.pathwren.workers.dev/crawler/google-cloudvertexbot.html) · [Google-CWS](https://www.pathwren.workers.dev/crawler/google-cws.html) · [Google-Extended](https://www.pathwren.workers.dev/crawler/google-extended.html) · [Google-GeminiNotebook](https://www.pathwren.workers.dev/crawler/google-gemininotebook.html) · [Google-InspectionTool](https://www.pathwren.workers.dev/crawler/google-inspectiontool.html) · [Google-Pinpoint](https://www.pathwren.workers.dev/crawler/google-pinpoint.html) · [Google-Read-Aloud](https://www.pathwren.workers.dev/crawler/google-read-aloud.html) · [Google-Safety](https://www.pathwren.workers.dev/crawler/google-safety.html) · [Google-Site-Verification](https://www.pathwren.workers.dev/crawler/google-site-verification.html) · [Googlebot](https://www.pathwren.workers.dev/crawler/googlebot.html) · [Googlebot-Image](https://www.pathwren.workers.dev/crawler/googlebot-image.html) · [Googlebot-News](https://www.pathwren.workers.dev/crawler/googlebot-news.html) · [Googlebot-Video](https://www.pathwren.workers.dev/crawler/googlebot-video.html) · [GoogleMessages](https://www.pathwren.workers.dev/crawler/googlemessages.html) · [GoogleOther](https://www.pathwren.workers.dev/crawler/googleother.html) · [GoogleOther-Image](https://www.pathwren.workers.dev/crawler/googleother-image.html) · [GoogleOther-Video](https://www.pathwren.workers.dev/crawler/googleother-video.html) · [GoogleProducer](https://www.pathwren.workers.dev/crawler/googleproducer.html) · [GPTBot](https://www.pathwren.workers.dev/crawler/gptbot.html) · [ia_archiver](https://www.pathwren.workers.dev/crawler/ia-archiver.html) · [ICC-Crawler](https://www.pathwren.workers.dev/crawler/icc-crawler.html) · [ImagesiftBot](https://www.pathwren.workers.dev/crawler/imagesiftbot.html) · [img2dataset](https://www.pathwren.workers.dev/crawler/img2dataset.html) · [ISSCyberRiskCrawler](https://www.pathwren.workers.dev/crawler/isscyberriskcrawler.html) · [Kagibot](https://www.pathwren.workers.dev/crawler/kagibot.html) · [KlaviyoAIBot](https://www.pathwren.workers.dev/crawler/klaviyoaibot.html) · [LAIONDownloader](https://www.pathwren.workers.dev/crawler/laiondownloader.html) · [Lightpanda](https://www.pathwren.workers.dev/crawler/lightpanda.html) · [Linguee Bot](https://www.pathwren.workers.dev/crawler/linguee-bot.html) · [Mediapartners-Google](https://www.pathwren.workers.dev/crawler/mediapartners-google.html) · [meta-externalagent](https://www.pathwren.workers.dev/crawler/meta-externalagent.html) · [meta-externalfetcher](https://www.pathwren.workers.dev/crawler/meta-externalfetcher.html) · [Meta-WebIndexer](https://www.pathwren.workers.dev/crawler/meta-webindexer.html) · [MistralAI-User](https://www.pathwren.workers.dev/crawler/mistralai-user.html) · [MojeekBot](https://www.pathwren.workers.dev/crawler/mojeekbot.html) · [OAI-SearchBot](https://www.pathwren.workers.dev/crawler/oai-searchbot.html) · [omgili](https://www.pathwren.workers.dev/crawler/omgili.html) · [omgilibot](https://www.pathwren.workers.dev/crawler/omgilibot.html) · [Panscient](https://www.pathwren.workers.dev/crawler/panscient.html) · [Perplexity-User](https://www.pathwren.workers.dev/crawler/perplexity-user.html) · [PerplexityBot](https://www.pathwren.workers.dev/crawler/perplexitybot.html) · [PetalBot](https://www.pathwren.workers.dev/crawler/petalbot.html) · [PhindBot](https://www.pathwren.workers.dev/crawler/phindbot.html) · [Pinterestbot](https://www.pathwren.workers.dev/crawler/pinterestbot.html) · [Poseidon Research Crawler](https://www.pathwren.workers.dev/crawler/poseidon-research-crawler.html) · [QualifiedBot](https://www.pathwren.workers.dev/crawler/qualifiedbot.html) · [QuillBot](https://www.pathwren.workers.dev/crawler/quillbot.html) · [Qwantbot](https://www.pathwren.workers.dev/crawler/qwantbot.html) · [Qwantbot-news](https://www.pathwren.workers.dev/crawler/qwantbot-news.html) · [Reflectionbot](https://www.pathwren.workers.dev/crawler/reflectionbot.html) · [SBIntuitionsBot](https://www.pathwren.workers.dev/crawler/sbintuitionsbot.html) · [Scrapy](https://www.pathwren.workers.dev/crawler/scrapy.html) · [Screaming Frog SEO Spider](https://www.pathwren.workers.dev/crawler/screaming-frog-seo-spider.html) · [SemrushBot-OCOB](https://www.pathwren.workers.dev/crawler/semrushbot-ocob.html) · [SeznamBot](https://www.pathwren.workers.dev/crawler/seznambot.html) · [ShapBot](https://www.pathwren.workers.dev/crawler/shapbot.html) · [Sidetrade indexer bot](https://www.pathwren.workers.dev/crawler/sidetrade-indexer-bot.html) · [Slackbot](https://www.pathwren.workers.dev/crawler/slackbot.html) · [Slackbot-LinkExpanding](https://www.pathwren.workers.dev/crawler/slackbot-linkexpanding.html) · [Storebot-Google](https://www.pathwren.workers.dev/crawler/storebot-google.html) · [TerraCotta](https://www.pathwren.workers.dev/crawler/terracotta.html) · [Thinkbot](https://www.pathwren.workers.dev/crawler/thinkbot.html) · [TikTokSpider](https://www.pathwren.workers.dev/crawler/tiktokspider.html) · [Timpibot](https://www.pathwren.workers.dev/crawler/timpibot.html) · [VelenPublicWebCrawler](https://www.pathwren.workers.dev/crawler/velenpublicwebcrawler.html) · [Webzio-Extended](https://www.pathwren.workers.dev/crawler/webzio-extended.html) · [wpbot](https://www.pathwren.workers.dev/crawler/wpbot.html) · [YaK](https://www.pathwren.workers.dev/crawler/yak.html) · [YandexAdditional](https://www.pathwren.workers.dev/crawler/yandexadditional.html) · [YandexAdditionalBot](https://www.pathwren.workers.dev/crawler/yandexadditionalbot.html) · [YandexBlogs](https://www.pathwren.workers.dev/crawler/yandexblogs.html) · [YandexBot](https://www.pathwren.workers.dev/crawler/yandexbot.html) · [YandexCalendar](https://www.pathwren.workers.dev/crawler/yandexcalendar.html) · [YandexComBot](https://www.pathwren.workers.dev/crawler/yandexcombot.html) · [YandexDirect](https://www.pathwren.workers.dev/crawler/yandexdirect.html) · [YandexFavicons](https://www.pathwren.workers.dev/crawler/yandexfavicons.html) · [YandexImages](https://www.pathwren.workers.dev/crawler/yandeximages.html) · [YandexMarket](https://www.pathwren.workers.dev/crawler/yandexmarket.html) · [YandexMedia](https://www.pathwren.workers.dev/crawler/yandexmedia.html) · [YandexMetrika](https://www.pathwren.workers.dev/crawler/yandexmetrika.html) · [YandexMobileBot](https://www.pathwren.workers.dev/crawler/yandexmobilebot.html) · [YandexRenderResourcesBot](https://www.pathwren.workers.dev/crawler/yandexrenderresourcesbot.html) · [YandexScreenshotBot](https://www.pathwren.workers.dev/crawler/yandexscreenshotbot.html) · [YandexVideo](https://www.pathwren.workers.dev/crawler/yandexvideo.html) · [YandexWebmaster](https://www.pathwren.workers.dev/crawler/yandexwebmaster.html) · [Yeti](https://www.pathwren.workers.dev/crawler/yeti.html) · [YouBot](https://www.pathwren.workers.dev/crawler/youbot.html)

## The file

[/robots/maximum-ai-visibility.txt](https://www.pathwren.workers.dev/robots/maximum-ai-visibility.txt) · [json](https://www.pathwren.workers.dev/policy/maximum-ai-visibility.json)

```text
# AI Crawler Index — policy: maximum-ai-visibility
# Maximum AI visibility
# Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines.
# Generated 2026-09-03 from https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html
# 133 crawlers named. Paste into robots.txt at your document root.

User-agent: AdsBot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

User-agent: AdsBot-Google-Mobile-Apps
Allow: /

User-agent: AI2Bot
Allow: /

User-agent: Ai2Bot-Dolma
Allow: /

User-agent: aiHitBot
Allow: /

User-agent: AIWebIndex
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Andibot
Allow: /

User-agent: Anomura
Allow: /

User-agent: anthropic-ai   # control token, no crawler uses this user-agent
Allow: /

User-agent: APIs-Google
Allow: /

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended   # control token, no crawler uses this user-agent
Allow: /

User-agent: archive.org_bot
Allow: /

User-agent: atlassian-bot
Allow: /

User-agent: AwarioRssBot
Allow: /

User-agent: AwarioSmartBot
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: bedrockbot
Allow: /

User-agent: bingbot
Allow: /

User-agent: Bytespider   # compliance disputed; enforce at the edge
Allow: /

User-agent: CCBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-Web   # control token, no crawler uses this user-agent
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Cloudflare-AutoRAG
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: cohere-training-data-crawler
Allow: /

User-agent: Cotoyogi
Allow: /

User-agent: Crawl4AI
Allow: /

User-agent: Crawlspace
Allow: /

User-agent: Diffbot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: EchoboxBot
Allow: /

User-agent: ExaSearchBot
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: facebookexternalhit
Allow: /

User-agent: Factset_spyderbot
Allow: /

User-agent: FeedFetcher-Google   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: FirecrawlAgent
Allow: /

User-agent: Google-Agent   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: Google-CWS   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Google-Extended   # control token, no crawler uses this user-agent
Allow: /

User-agent: Google-GeminiNotebook   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Google-InspectionTool
Allow: /

User-agent: Google-Pinpoint   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Google-Read-Aloud   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Google-Safety   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Google-Site-Verification   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-News
Allow: /

User-agent: Googlebot-Video
Allow: /

User-agent: GoogleMessages   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: GoogleOther-Image
Allow: /

User-agent: GoogleOther-Video
Allow: /

User-agent: GoogleProducer   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ia_archiver
Allow: /

User-agent: ICC-Crawler
Allow: /

User-agent: ImagesiftBot
Allow: /

User-agent: img2dataset
Allow: /

User-agent: ISSCyberRiskCrawler   # compliance disputed; enforce at the edge
Allow: /

User-agent: Kagibot
Allow: /

User-agent: KlaviyoAIBot
Allow: /

User-agent: LAIONDownloader   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: Lightpanda
Allow: /

User-agent: Linguee Bot   # compliance disputed; enforce at the edge
Allow: /

User-agent: Mediapartners-Google
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: Meta-WebIndexer
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: MojeekBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: omgili
Allow: /

User-agent: omgilibot
Allow: /

User-agent: panscient.com
Allow: /

User-agent: Perplexity-User   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: PetalBot
Allow: /

User-agent: PhindBot
Allow: /

User-agent: Pinterestbot
Allow: /

User-agent: Poseidon Research Crawler
Allow: /

User-agent: QualifiedBot
Allow: /

User-agent: QuillBot
Allow: /

User-agent: Qwantbot
Allow: /

User-agent: Qwantbot-news
Allow: /

User-agent: Reflectionbot
Allow: /

User-agent: SBIntuitionsBot
Allow: /

User-agent: Scrapy
Allow: /

User-agent: Screaming Frog SEO Spider
Allow: /

User-agent: SemrushBot-OCOB
Allow: /

User-agent: SeznamBot
Allow: /

User-agent: ShapBot
Allow: /

User-agent: Sidetrade indexer bot
Allow: /

User-agent: Slackbot
Allow: /

User-agent: Slackbot-LinkExpanding
Allow: /

User-agent: Storebot-Google
Allow: /

User-agent: TerraCotta
Allow: /

User-agent: Thinkbot   # compliance disputed; enforce at the edge
Allow: /

User-agent: TikTokSpider   # compliance disputed; enforce at the edge
Allow: /

User-agent: Timpibot
Allow: /

User-agent: VelenPublicWebCrawler
Allow: /

User-agent: Webzio-Extended
Allow: /

User-agent: wpbot
Allow: /

User-agent: YaK
Allow: /

User-agent: YandexAdditional
Allow: /

User-agent: YandexAdditionalBot
Allow: /

User-agent: YandexBlogs
Allow: /

User-agent: YandexBot
Allow: /

User-agent: YandexCalendar
Allow: /

User-agent: YandexComBot
Allow: /

User-agent: YandexDirect
Allow: /

User-agent: YandexFavicons
Allow: /

User-agent: YandexImages
Allow: /

User-agent: YandexMarket
Allow: /

User-agent: YandexMedia
Allow: /

User-agent: YandexMetrika   # operator states robots.txt does not apply; enforce at the edge
Allow: /

User-agent: YandexMobileBot
Allow: /

User-agent: YandexRenderResourcesBot
Allow: /

User-agent: YandexScreenshotBot
Allow: /

User-agent: YandexVideo
Allow: /

User-agent: YandexWebmaster
Allow: /

User-agent: Yeti
Allow: /

User-agent: YouBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.pathwren.workers.dev/sitemap.xml
```

## Sitemap

- [Full sitemap (XML)](https://www.pathwren.workers.dev/sitemap.xml) — every page, with dates
- [Full sitemap (markdown)](https://www.pathwren.workers.dev/sitemap.md) — the same map, readable
- [llms.txt](https://www.pathwren.workers.dev/llms.txt) — the whole host in one text file
- [documents.json](https://www.pathwren.workers.dev/documents.json) — every document, with its ETag
- [A2A agents](https://www.pathwren.workers.dev/a2a.html)
- [About and method](https://www.pathwren.workers.dev/about.html)
- [API](https://www.pathwren.workers.dev/api.html)
- [Changelog](https://www.pathwren.workers.dev/changelog.html)
- [Compliance](https://www.pathwren.workers.dev/compliance)
- [Contact](https://www.pathwren.workers.dev/contact)
- [Impressum · Anbieterkennzeichnung](https://www.pathwren.workers.dev/impressum)
- [AI Crawler Index](https://www.pathwren.workers.dev/index.html)
- [No model runs here](https://www.pathwren.workers.dev/inference.html)
- [Legal](https://www.pathwren.workers.dev/legal)
- [MCP server](https://www.pathwren.workers.dev/mcp-doctor.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-lint.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-netcheck.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-robots.html)
- [MCP transport: the GET and HEAD leg](https://www.pathwren.workers.dev/mcp-transport.html)
- [MCP server](https://www.pathwren.workers.dev/mcp-triage.html)
- [MCP server](https://www.pathwren.workers.dev/mcp.html)
- [Packages](https://www.pathwren.workers.dev/packages.html)
- [Pricing](https://www.pathwren.workers.dev/pricing)
- [Privacy](https://www.pathwren.workers.dev/privacy.html)
- [API reference](https://www.pathwren.workers.dev/reference)
- [Access, keys and sign-up](https://www.pathwren.workers.dev/register)
- [Security posture](https://www.pathwren.workers.dev/security.html)
- [Upstream status](https://www.pathwren.workers.dev/status.html)
- [Terms of use](https://www.pathwren.workers.dev/terms.html)
- [Trust](https://www.pathwren.workers.dev/trust)

## Machine copies of this page

- [HTML (canonical)](https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html)
- [JSON](https://www.pathwren.workers.dev/policy/maximum-ai-visibility.json)
- [Markdown](https://www.pathwren.workers.dev/policy/maximum-ai-visibility.md) — this document

This document is a markdown rendering of [https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html](https://www.pathwren.workers.dev/policy/maximum-ai-visibility.html), generated from that page's own bytes in the same build. The HTML page is canonical.