Which AI bots can crawl your site?
Check your robots.txt against 48 AI agents from 26 companies: ChatGPT, Claude, Perplexity, Gemini and more. See exactly who can read your content, who is blocked, and whether you are leaving AI visibility on the table.
Check your robots.txt.
Enter your URL and we'll fetch /robots.txt, parse it against every known AI user-agent, and show you what's allowed and what's blocked.
Let AI learn about your brand
Allow training bots like GPTBot, ClaudeBot, and Google-Extended to read your site. They feed the base models that answer questions about your category. Block them and the AI has no way to know you exist.
Stay visible in AI answers
Allow retrieval bots like OAI-SearchBot and PerplexityBot so users who ask ChatGPT about your category actually see you cited in the answer.
Three steps. Zero configuration.
Fetch
We pull /robots.txt from your domain with a neutral User-Agent. No JS execution, no tracking, no scraping beyond the one file.
Parse & match.
Your rules are matched against 48 known AI agents: training bots, retrieval bots and assistants. Each bot follows its own group, else the "*" group.
Score
We compute an AEO Readiness score based on whether the right bots are allowed, the wrong ones blocked, and your config is up to date.
Questions we get a lot.
Should I block GPTBot if I want to be visible in ChatGPT?
No — allow it. GPTBot trains the base model. If you block it, the AI has no way to know your brand exists when someone asks about your category. Allow training AND retrieval bots.
Does this check noindex meta tags too?
Not yet — this tool only parses /robots.txt. Full meta, header, and llms.txt coverage is part of the xSeek dashboard.
What's the AEO Readiness score?
A 0–100 composite: retrieval bots allowed (60%), training bots allowed (25%), file freshness (15%).
How often does the AI crawler list change?
Roughly once a quarter. We maintain the list from each vendor's docs and re-run our monitoring.
Is my URL stored?
Yes. Each scan is saved with the URL you checked, the result, your IP address and your browser user agent.
What if my site doesn't have a robots.txt?
We'll tell you that too — and show you exactly what to write, tuned to your AEO goals.