Robots.txt checker for AI crawlers
Enter your website address and see which AI crawlers your robots.txt allows or blocks, from OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple.
Try it
Free, no account. We do not save what you enter.
What this tool cannot tell you
robots.txt is a request, not a lock. This tool reads the file and says what it asks of each AI crawler. It cannot tell you whether a firewall turns a crawler away, whether an AI engine has read your pages, or whether any engine names your brand. The free audit checks the firewall and asks the engines.
To see whether the AI engines actually name your brand, run the free AEO audit.
Questions about this tool
Which AI crawlers does this check?
The answer and search crawlers of OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple, plus the training crawlers of Google, Apple, Meta and Common Crawl, and a few others. Each one is judged by the rules that name it, or by the rules for every crawler if none does.
Should I block GPTBot or ClaudeBot?
That is your choice. Training crawlers collect pages to train models, and search and answer crawlers fetch pages to answer a question. You can allow one kind and block the other. Blocking the search and answer crawlers can keep your pages out of those engines’ answers.
What if my site has no robots.txt?
Then every crawler may read everything, and that is not an error. Crawlers treat a missing robots.txt as no restrictions. You only need one if you want to ask some crawlers to stay away or to point them to your sitemap.
Does “allowed” mean AI will mention my brand?
No. Allowed means robots.txt does not ask the crawler to stay away. Whether an engine names you depends on what it finds and trusts. The free audit asks ChatGPT, Perplexity, Gemini and Google AI Overviews and shows you the real answers.