Which AI bots can crawl your site?
Analyze your robots.txt against the 14 AI crawlers that matter in 2026 — ChatGPT, Claude, Perplexity, Gemini, and more. See exactly who can read your content, who's blocked, and whether you're leaving AI visibility on the table.
Check your robots.txt.
Enter your URL and we'll fetch /robots.txt, parse it against every known AI user-agent, and show you what's allowed and what's blocked.
Let AI learn about your brand
Allow training bots like GPTBot, ClaudeBot, and Google-Extended to read your site. They feed the base models that answer questions about your category. Block them and the AI has no way to know you exist.
Stay visible in AI answers
Allow retrieval bots like OAI-SearchBot and PerplexityBot so users who ask ChatGPT about your category actually see you cited in the answer.
Three steps. Zero configuration.
Fetch
We pull /robots.txt from your domain with a neutral User-Agent. No JS execution, no tracking, no scraping beyond the one file.
Parse & match.
Your rules are matched against the 14 known AI crawlers — training bots, retrieval bots, and hybrid agents. Wildcards and precedence handled per RFC 9309.
Score
We compute an AEO Readiness score based on whether the right bots are allowed, the wrong ones blocked, and your config is up to date.
Questions we get a lot.
Should I block GPTBot if I want to be visible in ChatGPT?
No — allow it. GPTBot trains the base model. If you block it, the AI has no way to know your brand exists when someone asks about your category. Allow training AND retrieval bots.
Does this check noindex meta tags too?
Not yet — this tool only parses /robots.txt. Full meta, header, and llms.txt coverage is part of the xSeek dashboard.
What's the AEO Readiness score?
A 0–100 composite: retrieval bots allowed (60%), training bots allowed (25%), file freshness (15%).
How often does the AI crawler list change?
Roughly once a quarter. We maintain the list from each vendor's docs and re-run our monitoring.
Is my URL stored?
No. The scan is ephemeral — we don't log the URL, the result, or any headers your site returned.
What if my site doesn't have a robots.txt?
We'll tell you that too — and show you exactly what to write, tuned to your AEO goals.