AI Crawler Checker
Check if ChatGPT, Claude, and Perplexity can understand your site
This tool reads /robots.txt and /llms.txt. If direct CORS access fails, it retries through api.allorigins.win.
Quick guide
What can AI Crawler Checker help you do?
Free AI crawler checker for GPTBot, ClaudeBot, PerplexityBot, Googlebot, and llms.txt setup.
Best for
Use this for a preliminary check of a public site origin: whether robots.txt and llms.txt can be retrieved, and how simplified root-path rules apply to seven built-in crawler names. It does not crawl individual pages or establish actual crawler visits.
How to use it
Enter a website URL and start the check. Missing schemes become HTTPS, and page paths are discarded in favor of the origin. The widget requests /robots.txt and /llms.txt with timestamp parameters, retrying failed direct requests through a public proxy.
How to read the result
Found generally means a successful response; llms.txt must also be nonempty, but neither file is format-validated. Allowed or blocked describes only a simplified match for /. Exact user-agent groups take precedence over *, while unavailable robots content produces unknown results.
Privacy and limits
The browser contacts the target site and may send the requested file URL to api.allorigins.win after a direct failure. Both destinations may receive connection information. Avoid internal hosts, credentials in URLs, or site origins you should not disclose to a third party.
The parser does not fully support wildcard paths or end anchors, and cannot assess per-page restrictions, server defenses, or crawler compliance. A successful HTML fallback page can be reported as found. Inspect the original files before drawing conclusions about site-wide access.
Frequently Asked Questions
Why do some checks fail?
Some sites block browser fetches with CORS. That does not always mean crawlers are blocked.
What is GEO?
Generative Engine Optimization helps AI answer engines understand and cite your website.