website@ki-console.com
EN
EN DE FR IT ES UA

Crawler transparency

KI-Console-Bot

KI-Console-Bot is the crawler used for user-requested one-time AI readiness diagnostics.

What the bot fetches

  • robots.txt to understand crawler rules and opt-outs.
  • sitemap.xml when it is listed in robots.txt or available at the standard location.
  • llms.txt when it is available at the website root.
  • The homepage and only a few public pages at most for the submitted diagnostic.

Why we fetch it

A visitor or account owner submits a domain and asks KI-Console to run a one-time technical AI readiness diagnostic. The result is cached for public anonymous scans so the same site is not fetched repeatedly.

Identification

Requests identify themselves with this user-agent string:

KI-Console-Bot/1.0 (+https://ki-console.com/bot)

Rate limits

Anonymous public scans are cached for 24 hours per site. A live public scan is limited by IP quota and by the existing per-domain free-scan cooldown.

Opt out

To block public third-party KI-Console scans, add this group to your robots.txt. Owner-verified account scans are treated as authorized by the website owner.

robots.txt snippet

User-agent: KI-Console-Bot
Disallow: /