Live fetches
A person asks an AI system to read or verify a page, and the request happens at that moment. These visits are strong evidence of active demand.
Reference list
The AI crawler list 2026: all 59 agents
This list covers 59 AI and search crawlers according to vendor documentation (as of 08.10.2026), split into three groups: live fetches, training crawlers and AI search. It is generated directly from the KI-Console detection engine. User-triggered agents such as ChatGPT-User, Perplexity-User, Amzn-User and Google-Agent do not always follow robots.txt.
Training vs live vs search
The token alone is not enough. The group tells you what the visit probably means.
Training crawlers
Training agents collect public content on their own schedule. They can support future model knowledge, but they do not prove that a user asked about you today.
AI search crawlers
Search and index crawlers feed search systems that may power AI answers. Classic crawlability remains part of AI visibility.
Live fetches
Live fetches first 18 tokens
These agents usually appear when an AI product fetches a page for a concrete user action, answer, or research task.
| User-agent token | Known context |
|---|---|
ChatGPT-User |
OpenAI live fetch when a ChatGPT user asks for page access; according to OpenAI, robots.txt may not apply. Details |
OAI-SearchBot |
OpenAI search crawler for retrieval and search-backed answers. Details |
OAI-AdsBot |
OpenAI checks landing pages submitted as ads in ChatGPT; no training, robots.txt behavior not documented. Details |
Claude-User |
Anthropic live fetch when Claude reads a page for a user action. Details |
Claude-SearchBot |
Anthropic search crawler for search-backed Claude results. Details |
Perplexity-User |
Perplexity live fetch for a user’s question; according to Perplexity it generally ignores robots.txt. Details |
Meta-ExternalFetcher |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
MistralAI-User |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
DuckAssistBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Google-CloudVertexBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Kimi-User |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Manus-User |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
NotebookLM |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Google-GeminiNotebook |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Google-Agent |
Agents on Google infrastructure browse on a user’s behalf; Google lists it among user-triggered fetchers, which generally ignore robots.txt. Details |
Amzn-User |
Amazon live fetch for user requests such as Alexa queries; according to Amazon it may not follow all robots.txt directives. Details |
kagi-fetcher |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Gemini-Deep-Research |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Training
Training and model crawlers 28 tokens
These agents are associated with model training, data collection, or broad AI indexing. Their schedules and coverage are controlled by the provider.
| User-agent token | Known context |
|---|---|
GPTBot |
OpenAI training crawler for public web content access decisions. Details |
ClaudeBot |
Anthropic training crawler for Claude-related model data access. Details |
anthropic-ai |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Claude-Web |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Google-Extended |
Google control token for Gemini and Vertex AI use, not the classic Googlebot. Details |
Applebot-Extended |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
meta-externalagent |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Amazonbot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Bytespider |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
CCBot |
Common Crawl crawler; datasets from this ecosystem are widely used in AI training pipelines. Details |
cohere-training-data-crawler |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
cohere-ai |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
AI2Bot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Diffbot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
ImagesiftBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Webzio-Extended |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
DeepSeekBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
TongyiBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
YiyanBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
ChatGLM-Spider |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
bedrockbot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
PhindBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Bravebot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
omgili |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
PanguBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Timpibot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
YouBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
PetalBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
AI search
AI search and index crawlers 13 tokens
These crawlers build or refresh search indexes that can support AI answers, assistant search, or retrieval layers.
| User-agent token | Known context |
|---|---|
PerplexityBot |
Perplexity search crawler: surfaces and links websites in Perplexity results; according to Perplexity not used to train AI models. Details |
Amzn-SearchBot |
Amazon search crawler (for example for Alexa), no generative AI training; honors robots.txt. Details |
meta-webindexer |
Meta search crawler that improves the quality of Meta AI search results; a Disallow rule blocks it. Details |
Googlebot |
Google Search crawler; still relevant because AI answers often depend on search indexes. Details |
GoogleOther |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Bingbot |
Microsoft Bing crawler; relevant for Copilot and search-backed retrieval. Details |
DuckDuckBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
ExaBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
TavilyBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
LinkupBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
YandexAdditional |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
iAskBot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Applebot |
Known agent in this group; evaluate it through the group meaning and your log context. Details |
Copy-paste robots.txt starter
Use this as a conservative starter for the ten core agents KI-Console usually checks first. Adapt paths if private areas, checkout flows, or application endpoints must stay closed.
Curated core-agent group
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: Perplexity-User
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlHonest notes
- User-triggered fetches (ChatGPT-User, Perplexity-User, Amzn-User, Google-Agent) do not always follow robots.txt. Hard blocks need server or firewall rules. According to Anthropic, Claude-User honors robots.txt.
- The list changes constantly because providers rename, add, and retire agents.
- Grok and xAI currently have no documented crawler token in this engine, so they are not invented here.
- This page is generated directly from our detection engine.
- Vendor information as of 08.10.2026 (OpenAI, Anthropic, Google, Perplexity, Amazon, Meta). Names and behavior change; check the vendor page for important decisions.
Straight answers
What skeptical website owners usually ask first
KI-Console does not sell magic visibility. It makes your website more readable for AI systems and shows evidence where the tracker or logs provide it.
Why are GPTBot and ChatGPT-User separate?
GPTBot is associated with training and data collection. ChatGPT-User is a live fetch when a user action asks ChatGPT to read a page. They prove different things.
Should I allow every crawler on the list?
No. Use the list to make an intentional policy. Many public websites allow the important live and search agents, while sensitive or low-value areas stay blocked.
Where does the number :count come from?
The page counts the current KNOWN_CRAWLER_BOT_GROUPS entries in KI-Console. When the engine changes, this page changes with it.
Do all AI crawlers follow robots.txt?
No. According to their vendors, training and search crawlers such as GPTBot, ClaudeBot, PerplexityBot or Amzn-SearchBot honor it. User-triggered fetches such as ChatGPT-User, Perplexity-User, Amzn-User and Google-Agent do not always or not at all, because a person triggers them. According to Anthropic, Claude-User honors robots.txt.
Is PerplexityBot a training crawler?
No. According to Perplexity, PerplexityBot surfaces and links websites in search results and is not used to crawl content for AI foundation models. That is why it sits in the AI search group. Perplexity-User is the separate fetch when someone asks a question.
Check your own website
Find out which AI signals are working right now
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.