Training
ClaudeBot
Training crawler
ClaudeBot is the main Anthropic crawler associated with model training and broad public web access decisions.
Anthropic crawlers
Does Claude read your website?
Claude reads public pages through three Anthropic agents: ClaudeBot collects training data, Claude-User fetches a page when someone asks Claude about it, and Claude-SearchBot improves search. Anthropic states that all three honor robots.txt. If you block Claude-User, Claude can no longer fetch your page for user questions.
Anthropic
The Claude agents to separate
ClaudeBot, Claude-User, and Claude-SearchBot describe different access patterns. Separating them helps you see whether Claude can train from, fetch, or search your public pages.
Live fetches
Claude-User
Live Claude fetch
Claude-User appears when Claude fetches a page for a concrete user question. This is the live-read proof site owners usually want to see. According to Anthropic it honors robots.txt: a block prevents the fetch and may reduce visibility in user-directed web search.
AI search
Claude-SearchBot
Search-backed Claude access
According to Anthropic, Claude-SearchBot improves the quality of search results. Blocking it stops indexing for search and may reduce visibility and the accuracy of answers.
Legacy Anthropic tokens
Older or compatibility tokens such as anthropic-ai and Claude-Web can still matter in log analysis. KI-Console keeps them in the training group so old evidence is not lost.
anthropic-ai
Claude-Web
robots.txt guidance
Name the current Claude agents explicitly and include the legacy tokens if your policy is to allow Anthropic access. Keep private paths blocked as usual.
Claude starter rules
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /
Sitemap: https://www.example.com/sitemap.xmlWhy Claude access counts differently
Anthropic publishes no ranking list where you could earn a position; there is no ranking to improve. What counts is the fetch at the moment of the question, and there it only matters whether your page is reachable and contains the answer. There is nothing to buy and no head start through budget. Anyone who is reachable and clear starts from the same position as a corporation.
When customers hand Claude your page themselves
A large share of Claude visits do not come from search but from someone pasting a URL into a conversation: "what does this page say?", "compare these two offers". That is the most valuable visit of all, because there is a person with a concrete intent behind it. It fails, though, if the page rejects the fetch — and the customer then does not see an error from your server, only that Claude cannot say anything about it. That is exactly why keeping Claude-User reachable pays off.
What makes a page quotable
Claude names sources with a link when it draws on them. What gets quoted is what can be lifted out cleanly: a paragraph that stands on its own, with a clear statement and a number or a date in it. What is not quotable is a sentence that refers back to the previous section without repeating it. A practical test: copy one paragraph of your site into an empty file. Does it still make sense without context, and is it accurate? Then it is quotable.
Crawl-delay: here it actually works
Anthropic documents support for the Crawl-delay directive in robots.txt, with ClaudeBot as the example. That is unusual, as Google has ignored it for years. Anyone suffering from crawler load can slow the bot with a value in seconds instead of blocking it outright. That is almost always the better decision: a block costs visibility permanently, a delay only costs time. The directive goes in the same group as the other rules for that agent; Anthropic does not explicitly say whether Claude-User and Claude-SearchBot honor it too.
How KI-Console proves visits
KI-Console detects ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, and Claude-Web in logs and tracker reports, then groups them so live proof and training traffic stay separate.
What Claude access does not promise
Allowing Claude crawlers improves technical access but does not guarantee that Claude crawls, cites, recommends or ranks the website. An llms.txt does not change that: Anthropic has not committed to using it.
Straight answers
What skeptical website owners usually ask first
KI-Console does not sell magic visibility. It makes your website more readable for AI systems and shows evidence where the tracker or logs provide it.
Which Claude token proves a live read?
Claude-User is the clearest live-read token because it is tied to a concrete user action. ClaudeBot is a different training signal.
Should I still care about anthropic-ai and Claude-Web?
Yes for historical logs and compatibility. They may appear less often than current tokens, but keeping them mapped avoids missing older Anthropic evidence.
Does KI-Console show Claude traffic separately?
Yes. KI-Console recognises the Claude and legacy Anthropic tokens and separates live, training, and search traffic in the crawler proof workflow.
Do I have to allow ClaudeBot for Claude to read my page?
No. ClaudeBot is the training crawler; reading inside a conversation is Claude-User's job. You can disallow ClaudeBot and allow Claude-User — a common and well-founded position: the content should not go into training, but a person looking at the page right now should be allowed to see it. What matters is that the rules are written separately and no blanket block sits on top of them.
How do I tell a real Claude fetch from a fake one?
Not from the user agent alone — that can be forged in one line. Anthropic publishes the IP ranges of its agents; a fetch from outside those ranges is not a Claude fetch, whatever the identifier says. KI-Console checks against that instead of trusting the string. For your own check, a look at which network the request came from is enough.
Check your own website
Find out which AI signals are working right now
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.