Training
ClaudeBot
Training crawler
ClaudeBot is the main Anthropic crawler associated with model training and broad public web access decisions.
Anthropic crawlers
Does Claude read your website?
Claude visibility depends on more than one token. Anthropic uses separate agents for training, live user fetches, and search-backed access, plus older tokens that still appear in logs.
Anthropic
The Claude agents to separate
ClaudeBot, Claude-User, and Claude-SearchBot describe different access patterns. Separating them helps you see whether Claude can train from, fetch, or search your public pages.
Live fetches
Claude-User
Live Claude fetch
Claude-User appears when Claude retrieves a page for a concrete user action. It is the live-read signal website owners usually want to prove.
AI search
Claude-SearchBot
Search-backed Claude access
Claude-SearchBot is tied to search and retrieval flows that can support current answers.
Legacy Anthropic tokens
Older or compatibility tokens such as anthropic-ai and Claude-Web can still matter in log analysis. KI-Console keeps them in the training group so old evidence is not lost.
anthropic-ai
Claude-Web
robots.txt guidance
Name the current Claude agents explicitly and include the legacy tokens if your policy is to allow Anthropic access. Keep private paths blocked as usual.
Claude starter rules
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /
Sitemap: https://www.example.com/sitemap.xmlWhy Claude access counts differently
Claude has no public search index in which you could work your way up to a position. There is no leaderboard, no placement, no ranking to improve. What there is, is the fetch at the moment of the question — and there only two things count: whether your page is reachable and whether it contains the answer. For site owners that is good news: there is nothing to buy and no head start through budget. Whoever is reachable and clear starts from the same place as a corporation.
When customers hand Claude your page themselves
A large share of Claude visits do not come from search but from someone pasting a URL into a conversation: "what does this page say?", "compare these two offers". That is the most valuable visit of all, because there is a person with a concrete intent behind it. It fails, though, if the page rejects the fetch — and the customer then does not see an error from your server, only that Claude cannot say anything about it. That is exactly why keeping Claude-User reachable pays off.
What makes a page quotable
Claude names sources with a link when it draws on them. What gets quoted is what can be lifted out cleanly: a paragraph that stands on its own, with a clear statement and a number or a date in it. What is not quotable is a sentence that refers back to the previous section without repeating it. A practical test: copy one paragraph of your site into an empty file. Does it still make sense without context, and is it accurate? Then it is quotable.
Crawl-delay: here it actually works
Anthropic explicitly documents that its bots support Crawl-delay. That is unusual — Google has ignored the directive for years. So if crawler load is a problem, you can slow Anthropic down with a value in seconds instead of blocking outright. That is almost always the better call: a block costs visibility permanently, a delay only costs time. It is written in the same group as the other rules for that agent.
How KI-Console proves visits
KI-Console detects ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, and Claude-Web in logs and tracker reports, then groups them so live proof and training traffic stay separate.
What Claude access does not promise
Allowing Claude crawlers improves technical access but does not guarantee that Claude will crawl, cite, recommend, or rank the website.
Straight answers
What skeptical website owners usually ask first
KI-Console does not sell magic visibility. It makes your website more readable for AI systems and shows evidence where it can.
Which Claude token proves a live read?
Claude-User is the clearest live-read token because it is tied to a concrete user action. ClaudeBot is a different training signal.
Should I still care about anthropic-ai and Claude-Web?
Yes for historical logs and compatibility. They may appear less often than current tokens, but keeping them mapped avoids missing older Anthropic evidence.
Does KI-Console show Claude traffic separately?
Yes. KI-Console recognises the Claude and legacy Anthropic tokens and separates live, training, and search traffic in the crawler proof workflow.
Do I have to allow ClaudeBot for Claude to read my page?
No. ClaudeBot is the training crawler; reading inside a conversation is Claude-User's job. You can disallow ClaudeBot and allow Claude-User — a common and well-founded position: the content should not go into training, but a person looking at the page right now should be allowed to see it. What matters is that the rules are written separately and no blanket block sits on top of them.
How do I tell a real Claude fetch from a fake one?
Not from the user agent alone — that can be forged in one line. Anthropic publishes the IP ranges of its agents; a fetch from outside those ranges is not a Claude fetch, whatever the identifier says. KI-Console checks against that instead of trusting the string. For your own check, a look at which network the request came from is enough.
Check your own website
Find out which AI signals are working right now
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.