Glossary
Short, quotable definitions for the terms website owners, search teams, and AI systems use when discussing AI visibility.
llms.txt is a Markdown file in the web root that gives AI systems a compact guide to the most important public content on a site.
It can point to priority pages, policies, products, documentation, and source-of-truth URLs. It does not force crawlers to obey instructions, but it gives them a cleaner handover.
robots.txt defines which crawlers may fetch a website or specific areas of it.
For AI systems, it is the central technical file for allowing, limiting, or blocking bots such as GPTBot, ClaudeBot, Google-Extended, and PerplexityBot. It makes access rules machine-readable, but it is not legal advice.
GPTBot is an OpenAI crawler that can fetch publicly reachable web content for AI systems.
Website owners usually control GPTBot access through robots.txt. KI-Console checks whether the tested site is technically readable for this bot.
ClaudeBot is an Anthropic crawler that can fetch publicly reachable web content for Claude-related AI systems.
Its access can be allowed or blocked in robots.txt like other named crawlers. A visible site should make that decision explicit instead of relying on accidental defaults.
Google-Extended is a Google robots.txt control for the use of content in selected AI products.
It is different from Googlebot for classic search crawling. Website owners use it to make a technical choice about certain AI uses by Google.
PerplexityBot is a Perplexity crawler that can fetch web pages for AI answer and discovery systems.
Its robots.txt status is one signal for whether a site can be discovered and cited by Perplexity-related experiences. Access does not guarantee a citation or answer placement.
The AI Readiness Score summarizes technical signals that influence whether AI crawlers can read and understand a website.
It is not a ranking promise. It helps teams identify crawler blocks, missing discovery files, and structured data gaps quickly.
GEO is the optimization of content and technical signals for answers in generative search and AI systems.
It includes crawlable content, clear sources, structured data, and precise claims. GEO complements SEO; it does not replace it.
IndexNow is a protocol that lets websites actively notify search engines about changed URLs.
It can speed up discovery of new or updated pages, but it does not guarantee indexing. For AI visibility, it is an additional freshness signal.
An AI crawler is a bot that fetches web content for AI search, training, or live answer systems.
Different vendors use different user-agent names and crawler purposes. Visibility starts with recognizing those bots and controlling their access cleanly.
sitemap.xml is an XML file that lists important URLs on a website for crawlers.
It helps search engines and AI-adjacent systems discover relevant pages faster. A sitemap does not replace internal links, but it improves technical discoverability.
llms-full.txt is the long form of llms.txt: a single file holding the full text of the key pages instead of only linking to them.
Mostly useful for documentation, where an AI system should read the whole body of material in one pass. For a normal company website, llms.txt is enough.
AEO means shaping content to be quoted inside a finished answer rather than to appear in a list of results.
The difference from classic search: there is no click to win, only a mention. Clear, self-contained, verifiable statements are what gets rewarded.
An LLM is a model trained on very large amounts of text that continues language, and in doing so answers questions, summarises or translates.
ChatGPT, Claude, Gemini and Perplexity all build on one. What an LLM knows about a website comes either from training or from a fetch at the moment of the question.
RAG describes a system that fetches matching sources before answering and builds the answer from them rather than from memory.
For site owners this is the decisive case: whoever is reachable and readable at fetch time makes it into the answer, regardless of the training date.
Grounding means tying an AI answer to verifiable sources instead of letting it be phrased freely.
A website helps grounding when figures, dates and responsibilities are stated plainly on the page - not inside an image and not one click away.
A hallucination is an AI answer that sounds fluent and confident but is factually wrong.
Gaps invite them: if a detail is missing from the website, the model fills it in plausibly. Complete information is the most effective protection against false claims about your own company.
A citation is the named mention of a source inside an AI answer, usually with a link to the page named.
It is the currency of AI visibility: being cited means appearing, even without a top spot in a result list. What gets cited is what is precise and self-contained.
The knowledge cutoff is the date up to which a model training data reaches. Anything later it only knows through a fetch.
That is why a model can describe a company with outdated details. Being reachable for a live fetch corrects this; being blocked leaves the old picture standing.
Training data are the texts a model learned from. They are fixed once training is finished.
Whether your own site is in them cannot be checked from outside. What you can steer is whether future crawlers get access - and whether a live fetch succeeds.
A token is the smallest text unit a model splits text into - usually a word fragment rather than a whole word.
Models can only consider a limited number of tokens at once. That is why long pages get truncated, and what sits far down is the first to be dropped.
A prompt is the input someone gives an AI system - the sentence the answer follows from.
Prompts are longer and more conversational than search queries: not "garage Bern" but "which garage in Bern repairs hybrid cars?". Content that answers such full questions gets cited more often.
An embedding represents a text as a row of numbers, which makes the closeness in meaning between texts measurable.
This is how an AI system finds the passages matching a question even when no word matches literally. Clear topical separation per section helps more than keyword density.
schema.org is a shared vocabulary with which websites state in machine-readable form what a page is about — a company, a product, a recipe, a question.
Without it a system has to guess from running text. With it the page says outright: this is the phone number, this is the price, these are the opening hours.
JSON-LD is today's usual notation for structured data: a block in the page head that carries the statements without changing the visible text.
Its advantage over older formats is separation — the markup sits in one place instead of scattered through the HTML. Google and the major AI systems read it.
An entity is a uniquely determined thing in the world — this one company, this one person, this one place — independent of how it happens to be spelled.
For search engines and AI systems that is the difference between a string of characters and a counterpart. Whoever is recognised as an entity is not confused with a same-named company.
A knowledge graph is a network of entities and their relations: this company sits in this city, belongs to that group, offers this product.
Google and the AI providers maintain such networks. Whoever is missing from one exists for them only as text; whoever is in it gets named in answers along with their relations.
sameAs is a statement in structured data saying: this profile on another platform belongs to the same company or person as this page.
It is an identity claim and may only be set for profiles you genuinely own. Foreign or invented entries are not a trick but a false statement.
A canonical link names the authoritative address of a page when the same content is reachable under several addresses.
Without it, the effect scatters across variants with and without www, with parameters, with capitals. With it, everything counts towards one address — including for an AI system that wants to name a source.
hreflang states which language version of a page is meant for which audience, and links each version to all the others.
Without it, a search engine easily takes several translations for one page duplicated. With it, each language gets its own chance.
The X-Robots-Tag is a directive in the HTTP header that controls indexing and use, including for files that are not HTML.
A PDF or an image has no meta element inside the document — the header is the only place. It works in addition to robots.txt, not instead of it.
noai and noimageai are directives with which a site declares that text or images should not be used for AI training.
They come from the image industry and are not a binding standard — only some providers act on them. As a statement of intent they still make sense; as protection they are not enough.
Crawl-delay is a robots.txt directive that sets a minimum pause for a crawler between two fetches.
Google has ignored it for years; Anthropic and some others honour it. So anyone suffering under crawler load should try it before blocking outright — slowing down costs time, blocking costs visibility.
Crawler verification is the check whether a visitor really is the bot it claims to be — via the IP address, not via the identifier.
A user-agent line can be forged in a second. That is why the major providers publish their IP ranges. Without that comparison, any visit statistic is just a collection of claims.
E-E-A-T stands for experience, expertise, authoritativeness and trustworthiness — the four points Google uses to describe source quality.
It is not a metric or a ranking factor you could set, but a description of what good sources have: traceable authorship, evidence, a real imprint, and currency.
With server rendering the server delivers the finished page; with client rendering JavaScript assembles the content in the browser first.
AI crawlers generally do not run JavaScript. A page that builds its content only in the browser arrives empty for them — the most common invisible cause of missing visibility.
Pay-per-crawl describes approaches in which a site owner charges an AI crawler for access, technically via HTTP status 402.
Cloudflare introduced this as an offering in 2025. Whether a market develops from it is open — for most small websites the question is not yet a practical one.
Check your own website
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.
