Skip to content

Allowing our crawler

How to whitelist the AEOify scanner so it can access your site for accurate results.

Why you might need this

AEOify uses a real Chromium browser to render your pages — the same way AI search engines like Google AI Overviews and Perplexity crawl your site. Some Web Application Firewalls (WAFs) and CDN services may block automated browsers, which can result in incomplete or inaccurate scan results.

If your report shows unusually low scores or mentions a "Verifying your connection" page, your WAF is likely blocking our crawler.

How to identify our crawler

Our crawler sends a standard Chrome user-agent with an AEOifyBot identifier appended:

AEOifyBot/1.0 (+https://aeoify.ai/help/allowing-our-crawler)

The full user-agent string is:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 AEOifyBot/1.0 (+https://aeoify.ai/help/allowing-our-crawler)

You can whitelist the crawler by either:

  • User-agent string — match on the AEOifyBot substring
  • IP address — all requests originate from a single static IP (contact support@aeoify.ai for the current address)

Cloudflare

Most sites behind Cloudflare can allow our crawler with a single WAF rule:

  1. Log in to your Cloudflare dashboard
  2. Select your domain
  3. Go to Security > WAF > Custom rules
  4. Click Create rule
  5. Set the rule name to Allow AEOifyBot
  6. Under When incoming requests match, choose:
    • Field: User Agent
    • Operator: contains
    • Value: AEOifyBot
  7. Set Then take action to Skip > check All remaining custom rules
  8. Click Deploy

The crawler should now pass through on your next scan.

Shopify stores

Shopify uses Cloudflare by default for all stores. If your Shopify site is blocking our crawler, follow the Cloudflare steps above using your Cloudflare dashboard. If you do not have a Cloudflare account (Shopify manages it for you), you may need to contact Shopify support to whitelist the AEOifyBot user-agent.

Other WAFs

The general approach is the same for any WAF — create a rule that allows requests where the user-agent contains AEOifyBot.

  • AWS WAF — Add a string match condition on the User-Agent header for AEOifyBot, then create a rule that allows matching requests.
  • Sucuri — Go to Settings > Whitelist, add the user-agent AEOifyBot.
  • Wordfence (WordPress) — Go to Firewall > Blocking, add a user-agent pattern allowlist for AEOifyBot.

If your WAF supports IP-based allowlisting and you prefer that approach, contact us at support@aeoify.ai for our static IP address.

Verify it worked

After adding the whitelist rule:

  1. Go to your Dashboard and find the report for your site
  2. Click Rescan to run a fresh evaluation
  3. Compare the new scores against your previous report — if the crawler can now access your pages, you should see more detailed recommendations and potentially different scores

Related articles