Short answer
meta-webindexer analyzes content to improve the quality of Meta AI search results.
AI crawler guide
meta-webindexer: set robots.txt correctly
meta-webindexer analyzes content to improve the quality of Meta AI search results.
Crawler facts
Facts about meta-webindexer
User-agent token
meta-webindexer
Operator
Meta
KI-Console group
AI search
Search or index crawler: sitemap, crawlability, and classic SEO still matter.
robots.txt
Yes, verify in logs
Documentation
Open source documentationWhat is meta-webindexer?
Meta separates it from the training and indexing crawler meta-externalagent and the user-triggered meta-externalfetcher. The user agent appears as meta-webindexer/1.1.
Purpose and SEO meaning
The KI-Console group is search. If you allow it, Meta AI can cite and link your content.
Visibility and privacy note
Blocking meta-webindexer can reduce how systems from Meta see, index, train on, or retrieve your public content. Allowing it improves technical access, but it never guarantees crawling, ranking, citation, or a positive AI answer.
Meta lists no robots.txt exception for meta-webindexer; a Disallow rule blocks it. Blocking meta-externalagent does not affect it.
robots.txt for meta-webindexer
Use the exact token meta-webindexer in your robots.txt. Keep private paths blocked separately if only part of the site should be available.
Allow :bot
User-agent: meta-webindexer
Allow: /Block :bot
User-agent: meta-webindexer
Disallow: /Straight answers
What skeptical website owners usually ask first
KI-Console does not sell magic visibility. It makes your website more readable for AI systems and shows evidence where the tracker or logs provide it.
What is meta-webindexer?
meta-webindexer is a crawler or robots.txt control token operated by Meta. The facts above show its purpose, robots.txt behavior, and exact allow or block rules.
How do I block meta-webindexer in robots.txt?
Use User-agent: meta-webindexer and Disallow: /. robots.txt is a directive for compliant crawlers, not a technical access barrier.
Does blocking meta-webindexer reduce AI visibility?
It can reduce visibility in systems from Meta if those systems can no longer crawl, index, train on, or retrieve your content. The effect depends on the bot type and product.
Check your own website
Find out which AI signals are working right now
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.