# AI Crawler Index — user-agent substrings, one per line, 2026-09-03
# https://www.pathwren.workers.dev/data/user-agents.txt
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
AhrefsBot
AhrefsSiteAudit
AI2Bot
Ai2Bot-Dolma
aiHitBot
AIWebIndex
Amazonbot
Andibot
Anomura
anthropic-ai
APIs-Google
Applebot
archive.org_bot
atlassian-bot
AwarioRssBot
AwarioSmartBot
Baiduspider
barkrowler
bedrockbot
bingbot
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
DataForSeoBot
Diffbot
dotbot
DuckAssistBot
DuckDuckBot
EchoboxBot
ExaSearchBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FeedFetcher-Google
FirecrawlAgent
Google-Agent
Google-CloudVertexBot
Google-CWS
Google-GeminiNotebook
Google-InspectionTool
Google-Pinpoint
Google-Read-Aloud
Google-Safety
Google-Site-Verification
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
GoogleMessages
GoogleOther
GoogleOther-Image
GoogleOther-Video
GoogleProducer
GPTBot
ia_archiver
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kagibot
KlaviyoAIBot
LAIONDownloader
Lightpanda
Linguee Bot
Mediapartners-Google
meta-externalagent
meta-externalfetcher
Meta-WebIndexer
MistralAI-User
MJ12bot
MojeekBot
OAI-SearchBot
omgili
omgilibot
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Pinterestbot
Poseidon Research Crawler
QualifiedBot
QuillBot
Qwantbot
Qwantbot-news
Reflectionbot
rogerbot
SBIntuitionsBot
Scrapy
Screaming Frog SEO Spider
SemrushBot
SemrushBot-BA
SemrushBot-ESI
SemrushBot-FT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
SEOkicks
serpstatbot
SeznamBot
ShapBot
Sidetrade indexer bot
SiteAuditBot
Slackbot
Slackbot-LinkExpanding
SplitSignalBot
Storebot-Google
TerraCotta
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YandexBlogs
YandexBot
YandexCalendar
YandexComBot
YandexDirect
YandexFavicons
YandexImages
YandexMarket
YandexMedia
YandexMetrika
YandexMobileBot
YandexRenderResourcesBot
YandexScreenshotBot
YandexVideo
YandexWebmaster
Yeti
YouBot