Robots.txt generator for AI crawlers
Choose which AI crawlers may read your site, add your sitemap, and copy or download a robots.txt that does exactly that.
Try it
Free, no account. We do not save what you enter.
What this tool cannot tell you
This tool writes a robots.txt from your choices. It does not edit your site or replace your current file, so compare it with the one you have. robots.txt is a request that well-behaved crawlers follow, not a lock, and it cannot make an AI engine mention you.
To see whether the AI engines actually name your brand, run the free AEO audit.
Questions about this tool
Which crawlers does this generator block?
Only the AI crawlers you choose to block, in three groups: answer crawlers, training crawlers and others. It never blocks Googlebot, Bingbot or Applebot, so blocking AI crawlers here does not take you out of ordinary search.
Should I block AI answer crawlers?
Usually not. They fetch pages so an AI engine can answer a question, so blocking them can keep your pages out of those answers. Training crawlers collect pages to train models, and blocking them is a business choice.
Where do I put the file?
Save it as robots.txt in the top folder of your website, so it opens at yourcompany.com/robots.txt. If you already have one, compare the two first, because this file replaces everything in it. Then check it with the robots.txt checker.
What are the paths for?
A path such as /admin/ asks every crawler to stay out of that part of your site. List one per line, starting with a slash. A path of just a slash is refused, because it would block your whole site.