Our crawler

AEOCornerBot

AEOCornerBot is the crawler AEO Corner uses to read a website when someone runs a free audit or scans their own site. This page explains what it does and how to stop it.

How to recognise it

Every request names itself with this user agent:

AEOCornerBot/1.0 (+https://aeocorner.com/bot)

What it does

AEOCornerBot reads a few public pages to work out how easy your site is for AI engines to find and quote.

  • It reads public pages only: your home page and a handful of key pages, plus robots.txt and your sitemap. It never signs in, fills in forms or guesses addresses.
  • It checks whether AI search crawlers (such as GPTBot, ClaudeBot and PerplexityBot) are allowed to read your site, and whether your pages are easy for AI to read and quote. The result goes to the person who asked for the audit.
  • It does not copy your site for resale and does not train AI models.

How politely it behaves

  • At most 2 requests at a time to your site, at least half a second apart.
  • A free audit reads at most a few dozen pages and runs once per website per day at most.
  • It follows robots.txt. If robots.txt can’t be fetched because your server is failing, it stays away rather than guess.

The one exception is a signed-in customer scanning their own website: that scan can read pages robots.txt would otherwise hide from our bot, and the report says so. The free audit never does this.

How to stop it

Add this to your robots.txt:

User-agent: AEOCornerBot
Disallow: /

Our audit will then say it was asked not to read your site, rather than scoring pages it didn’t see. To block only part of your site, list those paths under Disallow instead.

Questions or a problem

If AEOCornerBot is doing something you don’t expect, write to [email protected] with your website address and the time you saw it. We’ll look at it and reply.

Free AEO audit

See what AI says about your brand

Enter your website and get a real report in about five minutes: your AEO Score, the actual answers AI gave, and your top five fixes.

Compare with a competitor (optional)

Free · about 5 minutes · no account or credit card needed.