# AI Crawler Index — one case-insensitive regex matching every crawler, 2026-09-03
(AdsBot\-Google|AdsBot\-Google\-Mobile|AdsBot\-Google\-Mobile\-Apps|AhrefsBot|AhrefsSiteAudit|AI2Bot|Ai2Bot\-Dolma|aiHitBot|AIWebIndex|Amazonbot|Andibot|Anomura|anthropic\-ai|APIs\-Google|Applebot|archive\.org_bot|atlassian\-bot|AwarioRssBot|AwarioSmartBot|Baiduspider|barkrowler|bedrockbot|bingbot|Bytespider|CCBot|ChatGPT\ Agent|ChatGPT\-User|Claude\-SearchBot|Claude\-User|Claude\-Web|ClaudeBot|Cloudflare\-AutoRAG|cohere\-ai|cohere\-training\-data\-crawler|Cotoyogi|Crawl4AI|Crawlspace|DataForSeoBot|Diffbot|dotbot|DuckAssistBot|DuckDuckBot|EchoboxBot|ExaSearchBot|FacebookBot|facebookexternalhit|Factset_spyderbot|FeedFetcher\-Google|FirecrawlAgent|Google\-Agent|Google\-CloudVertexBot|Google\-CWS|Google\-GeminiNotebook|Google\-InspectionTool|Google\-Pinpoint|Google\-Read\-Aloud|Google\-Safety|Google\-Site\-Verification|Googlebot|Googlebot\-Image|Googlebot\-News|Googlebot\-Video|GoogleMessages|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GoogleProducer|GPTBot|ia_archiver|ICC\-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kagibot|KlaviyoAIBot|LAIONDownloader|Lightpanda|Linguee\ Bot|Mediapartners\-Google|meta\-externalagent|meta\-externalfetcher|Meta\-WebIndexer|MistralAI\-User|MJ12bot|MojeekBot|OAI\-SearchBot|omgili|omgilibot|panscient\.com|Perplexity\-User|PerplexityBot|PetalBot|PhindBot|Pinterestbot|Poseidon\ Research\ Crawler|QualifiedBot|QuillBot|Qwantbot|Qwantbot\-news|Reflectionbot|rogerbot|SBIntuitionsBot|Scrapy|Screaming\ Frog\ SEO\ Spider|SemrushBot|SemrushBot\-BA|SemrushBot\-ESI|SemrushBot\-FT|SemrushBot\-OCOB|SemrushBot\-SI|SemrushBot\-SWA|SEOkicks|serpstatbot|SeznamBot|ShapBot|Sidetrade\ indexer\ bot|SiteAuditBot|Slackbot|Slackbot\-LinkExpanding|SplitSignalBot|Storebot\-Google|TerraCotta|Thinkbot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio\-Extended|wpbot|YaK|YandexAdditional|YandexAdditionalBot|YandexBlogs|YandexBot|YandexCalendar|YandexComBot|YandexDirect|YandexFavicons|YandexImages|YandexMarket|YandexMedia|YandexMetrika|YandexMobileBot|YandexRenderResourcesBot|YandexScreenshotBot|YandexVideo|YandexWebmaster|Yeti|YouBot)