Live fetches
A person asks an AI system to read or verify a page, and the request happens at that moment. These visits are strong evidence of active demand.
Reference list
The AI crawler list 2026: all 53 agents
This page is generated directly from the KI-Console detection engine. It separates real user-triggered live fetches from training crawlers and AI search crawlers, so crawler proof does not mix different signals.
Training vs live vs search
The token alone is not enough. The group tells you what the visit probably means.
Training crawlers
Training agents collect public content on their own schedule. They can support future model knowledge, but they do not prove that a user asked about you today.
AI search crawlers
Search and index crawlers feed search systems that may power AI answers. Classic crawlability remains part of AI visibility.
Live fetches
Live fetches first 14 tokens
These agents usually appear when an AI product fetches a page for a concrete user action, answer, or research task.
| User-agent token | Known context |
|---|---|
ChatGPT-User |
OpenAI live fetch when a ChatGPT user asks for page access. |
OAI-SearchBot |
OpenAI search crawler for retrieval and search-backed answers. |
Claude-User |
Anthropic live fetch when Claude reads a page for a user action. |
Claude-SearchBot |
Anthropic search crawler for search-backed Claude results. |
Perplexity-User |
Perplexity live user fetch for answer generation or page checks. |
Meta-ExternalFetcher |
Known agent in this group; evaluate it through the group meaning and your log context. |
MistralAI-User |
Known agent in this group; evaluate it through the group meaning and your log context. |
DuckAssistBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Google-CloudVertexBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Kimi-User |
Known agent in this group; evaluate it through the group meaning and your log context. |
Manus-User |
Known agent in this group; evaluate it through the group meaning and your log context. |
NotebookLM |
Known agent in this group; evaluate it through the group meaning and your log context. |
kagi-fetcher |
Known agent in this group; evaluate it through the group meaning and your log context. |
Gemini-Deep-Research |
Known agent in this group; evaluate it through the group meaning and your log context. |
Training
Training and model crawlers 29 tokens
These agents are associated with model training, data collection, or broad AI indexing. Their schedules and coverage are controlled by the provider.
| User-agent token | Known context |
|---|---|
GPTBot |
OpenAI training crawler for public web content access decisions. |
ClaudeBot |
Anthropic training crawler for Claude-related model data access. |
anthropic-ai |
Known agent in this group; evaluate it through the group meaning and your log context. |
Claude-Web |
Known agent in this group; evaluate it through the group meaning and your log context. |
Google-Extended |
Google control token for Gemini and Vertex AI use, not the classic Googlebot. |
Applebot-Extended |
Known agent in this group; evaluate it through the group meaning and your log context. |
meta-externalagent |
Known agent in this group; evaluate it through the group meaning and your log context. |
Amazonbot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Bytespider |
Known agent in this group; evaluate it through the group meaning and your log context. |
CCBot |
Common Crawl crawler; datasets from this ecosystem are widely used in AI training pipelines. |
PerplexityBot |
Perplexity crawler for index and answer-system discovery. |
cohere-training-data-crawler |
Known agent in this group; evaluate it through the group meaning and your log context. |
cohere-ai |
Known agent in this group; evaluate it through the group meaning and your log context. |
AI2Bot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Diffbot |
Known agent in this group; evaluate it through the group meaning and your log context. |
ImagesiftBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Webzio-Extended |
Known agent in this group; evaluate it through the group meaning and your log context. |
DeepSeekBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
TongyiBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
YiyanBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
ChatGLM-Spider |
Known agent in this group; evaluate it through the group meaning and your log context. |
bedrockbot |
Known agent in this group; evaluate it through the group meaning and your log context. |
PhindBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Bravebot |
Known agent in this group; evaluate it through the group meaning and your log context. |
omgili |
Known agent in this group; evaluate it through the group meaning and your log context. |
PanguBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Timpibot |
Known agent in this group; evaluate it through the group meaning and your log context. |
YouBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
PetalBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
AI search
AI search and index crawlers 10 tokens
These crawlers build or refresh search indexes that can support AI answers, assistant search, or retrieval layers.
| User-agent token | Known context |
|---|---|
Googlebot |
Google Search crawler; still relevant because AI answers often depend on search indexes. |
GoogleOther |
Known agent in this group; evaluate it through the group meaning and your log context. |
Bingbot |
Microsoft Bing crawler; relevant for Copilot and search-backed retrieval. |
DuckDuckBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
ExaBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
TavilyBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
LinkupBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
YandexAdditional |
Known agent in this group; evaluate it through the group meaning and your log context. |
iAskBot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Applebot |
Known agent in this group; evaluate it through the group meaning and your log context. |
Copy-paste robots.txt starter
Use this as a conservative starter for the ten core agents KI-Console usually checks first. Adapt paths if private areas, checkout flows, or application endpoints must stay closed.
Curated core-agent group
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: Perplexity-User
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlHonest notes
- The list changes constantly because providers rename, add, and retire agents.
- Grok and xAI currently have no documented crawler token in this engine, so they are not invented here.
- This page is generated directly from our detection engine.
Straight answers
What skeptical website owners usually ask first
KI-Console does not sell magic visibility. It makes your website more readable for AI systems and shows evidence where it can.
Why are GPTBot and ChatGPT-User separate?
GPTBot is associated with training and data collection. ChatGPT-User is a live fetch when a user action asks ChatGPT to read a page. They prove different things.
Should I allow every crawler on the list?
No. Use the list to make an intentional policy. Many public websites allow the important live and search agents, while sensitive or low-value areas stay blocked.
Where does the number :count come from?
The page counts the current KNOWN_CRAWLER_BOT_GROUPS entries in KI-Console. When the engine changes, this page changes with it.
Check your own website
Find out which AI signals are working right now
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.