The full list of AI crawler signatures we match.

We match 60+ answer-engine crawlers. This page is the source of truth. If your logs show one we miss, tell us — we’ll add it within 7 days and credit you.

68 signatures currently active. First-seen dates are not tracked per signature, so the table shows the date each entry was last verified against live traffic.

CrawlerSignature typeExample payloadWhat it doesLast verifiedChangelog
chatgptuser-agent tokenGPTBotOpenAI training crawler2026-09-03entry
chatgptuser-agent tokenOAI-SearchBotOpenAI search index crawler2026-09-03entry
chatgptuser-agent tokenChatGPT-UserChatGPT live browsing on user request2026-09-03entry
chatgptuser-agent tokenOpenAI-ImageOpenAI image fetcher2026-09-03entry
chatgptuser-agent tokenopenai.comGeneric OpenAI agent string2026-09-03entry
claudeuser-agent tokenClaudeBotAnthropic crawler2026-09-03entry
claudeuser-agent tokenClaude-WebAnthropic web fetcher2026-09-03entry
claudeuser-agent tokenanthropic-aiAnthropic legacy agent2026-09-03entry
claudeuser-agent tokenClaude-UserClaude live browsing2026-09-03entry
claudeuser-agent tokenClaude-SearchBotClaude search index crawler2026-09-03entry
copilotuser-agent tokenMicrosoftPreviewMicrosoft Copilot preview fetcher2026-09-03entry
copilotuser-agent tokenmsnbot-mediaMicrosoft media fetcher2026-09-03entry
copilotuser-agent tokenBingPreviewBing preview fetcher2026-09-03entry
copilotuser-agent tokenBingBot-AIBing AI answer crawler2026-09-03entry
copilotuser-agent tokenCopilotMicrosoft Copilot agent2026-09-03entry
geminiuser-agent tokenGoogle-CloudVertex-BotVertex AI crawler2026-09-03entry
geminiuser-agent tokenGoogleOtherGoogle experimental/AI crawler2026-09-03entry
geminiuser-agent tokenGoogleAgent-MarinerGoogle agentic browsing2026-09-03entry
geminiuser-agent tokenGoogle-FirebaseGoogle AI product fetcher2026-09-03entry
geminiuser-agent tokenGoogle-ExtendedGoogle Gemini training opt-in token2026-09-03entry
geminiuser-agent tokenGoogle-CloudVertexBotVertex AI grounding fetcher2026-09-03entry
otheruser-agent tokenNeevaBotNeeva legacy AI crawler2026-09-03entry
otheruser-agent tokenAndibotAndi search engine2026-09-03entry
otheruser-agent tokenPhindbotPhind answer engine2026-09-03entry
otheruser-agent tokenPoeQuora Poe agent2026-09-03entry
otheruser-agent tokenApplebot-ExtendedApple Intelligence training crawler2026-09-03entry
otheruser-agent tokenApplebotApple search/AI crawler2026-09-03entry
otheruser-agent tokencohere-aiCohere crawler2026-09-03entry
otheruser-agent tokencohere-training-data-crawlerCohere training crawler2026-09-03entry
otheruser-agent tokenMeta-ExternalAgentMeta AI crawler2026-09-03entry
otheruser-agent tokenMeta-ExternalFetcherMeta AI live fetcher2026-09-03entry
otheruser-agent tokenFacebookBotMeta training crawler2026-09-03entry
otheruser-agent tokenAmazonbotAmazon Alexa/AI crawler2026-09-03entry
otheruser-agent tokenBytespiderByteDance/Doubao crawler2026-09-03entry
otheruser-agent tokenTikTokSpiderByteDance crawler2026-09-03entry
otheruser-agent tokenCCBotCommon Crawl (LLM training corpus)2026-09-03entry
otheruser-agent tokenai2botAllen Institute crawler2026-09-03entry
otheruser-agent tokenAI2Bot-DolmaAllen Institute Dolma corpus2026-09-03entry
otheruser-agent tokenYouBotYou.com answer engine2026-09-03entry
otheruser-agent tokenDiffbotDiffbot knowledge graph2026-09-03entry
otheruser-agent tokenImagesiftBotImagesift/Hive AI crawler2026-09-03entry
otheruser-agent tokenPetalBotHuawei Petal AI search2026-09-03entry
otheruser-agent tokenDataForSeoBotDataForSEO crawler2026-09-03entry
otheruser-agent tokenSemrushBot-OCOBSemrush AI content crawler2026-09-03entry
otheruser-agent tokenTimpibotTimpi decentralized index2026-09-03entry
otheruser-agent tokenomgiliWebz.io LLM data crawler2026-09-03entry
otheruser-agent tokenomgilibotWebz.io crawler2026-09-03entry
otheruser-agent tokenWebzio-ExtendedWebz.io AI dataset crawler2026-09-03entry
otheruser-agent tokenICC-CrawlerNICT AI research crawler2026-09-03entry
otheruser-agent tokenKangaroo BotKangaroo LLM crawler2026-09-03entry
otheruser-agent tokenimg2datasetLAION dataset builder2026-09-03entry
otheruser-agent tokenScrapyGeneric scraping framework2026-09-03entry
otheruser-agent tokenpython-requestsGeneric scripted fetch2026-09-03entry
otheruser-agent tokennode-fetchGeneric scripted fetch2026-09-03entry
otheruser-agent tokenGo-http-clientGeneric scripted fetch2026-09-03entry
otheruser-agent tokenMistralAI-UserMistral Le Chat browsing2026-09-03entry
otheruser-agent tokenMistral-CrawlerMistral crawler2026-09-03entry
otheruser-agent tokenxAI-BotxAI Grok crawler2026-09-03entry
otheruser-agent tokenGrokBotxAI Grok agent2026-09-03entry
otheruser-agent tokenDeepSeekBotDeepSeek crawler2026-09-03entry
otheruser-agent tokenQwenBotAlibaba Qwen crawler2026-09-03entry
otheruser-agent tokenSentiBotSenti AI crawler2026-09-03entry
otheruser-agent tokenFirecrawlFirecrawl agent scraper2026-09-03entry
otheruser-agent tokenBrightbotBright Data AI crawler2026-09-03entry
otheruser-agent tokenLinerBotLiner answer engine2026-09-03entry
perplexityuser-agent tokenPerplexity-aiPerplexity legacy agent2026-09-03entry
perplexityuser-agent tokenPerplexity-UserPerplexity live user fetch2026-09-03entry
perplexityuser-agent tokenPerplexityBotPerplexity index crawler2026-09-03entry

How we classify

Classification runs server-side on every request, before any JavaScript executes, and in three passes. The first pass looks for an exact documented user-agent token such as the ones OpenAI, Anthropic, Perplexity and Google publish. A token match is the strongest signal available and is used directly. The second pass applies a regular expression for known variants — version suffixes, vendor-specific fetcher strings, and agents that embed a product name inside a generic library string.

The third pass handles requests whose agent string is weak, generic or absent by checking the source address against published crawler IP ranges. That pass also runs as a cross-check on token matches where the vendor publishes ranges, because a request claiming a crawler identity from outside the vendor’s network is an impersonator and is reported as such. Nothing in this pipeline reads or writes anything on the visitor’s device, which is why it works without cookies or a consent banner.

Frequently asked questions

How do you decide whether a request is an AI crawler?

We match the user-agent against a documented token first, then against a regex for variants, and finally against published IP ranges when the agent string is weak or absent.

What if a crawler is missing from this list?

Send us the raw log line. We add verified signatures within seven days and credit the reporter in the changelog entry.

Do you block these crawlers?

No. Citealytics only classifies and reports them. Whether to allow a crawler is a robots.txt decision you make on your own site.

Can someone fake a crawler user-agent?

Yes, which is why token matches are cross-checked against published IP ranges where the vendor provides them. A token match from outside those ranges is treated as an impersonator.

Measure the AI traffic your analytics is hiding.

One line of script, no cookies, no consent banner. Citealytics classifies every visit against a registry of answer-engine signatures and shows which pages get cited.

Start free — 10k events/mo →

Related reading