Allowing our crawler
How to whitelist the AEOify scanner so it can access your site for accurate results.
Why you might need this
AEOify uses a real Chromium browser to render your pages — the same way AI search engines like Google AI Overviews and Perplexity crawl your site. Some Web Application Firewalls (WAFs) and CDN services may block automated browsers, which can result in incomplete or inaccurate scan results.
If your report shows unusually low scores or mentions a "Verifying your connection" page, your WAF is likely blocking our crawler.
How to identify our crawler
Our crawler sends a standard Chrome user-agent with an AEOifyBot identifier appended:
AEOifyBot/1.0 (+https://aeoify.ai/help/allowing-our-crawler)
The full user-agent string is:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 AEOifyBot/1.0 (+https://aeoify.ai/help/allowing-our-crawler)
You can whitelist the crawler by either:
- User-agent string — match on the
AEOifyBotsubstring - IP address — all requests originate from a single static IP (contact support@aeoify.ai for the current address)
Cloudflare
Most sites behind Cloudflare can allow our crawler with a single WAF rule:
- Log in to your Cloudflare dashboard
- Select your domain
- Go to Security > WAF > Custom rules
- Click Create rule
- Set the rule name to
Allow AEOifyBot - Under When incoming requests match, choose:
- Field: User Agent
- Operator: contains
- Value:
AEOifyBot
- Set Then take action to Skip > check All remaining custom rules
- Click Deploy
The crawler should now pass through on your next scan.
Shopify stores
Shopify uses Cloudflare by default for all stores. If your Shopify site is blocking our crawler, follow the Cloudflare steps above using your Cloudflare dashboard. If you do not have a Cloudflare account (Shopify manages it for you), you may need to contact Shopify support to whitelist the AEOifyBot user-agent.
Other WAFs
The general approach is the same for any WAF — create a rule that allows requests where the user-agent contains AEOifyBot.
- AWS WAF — Add a string match condition on the
User-Agentheader forAEOifyBot, then create a rule that allows matching requests. - Sucuri — Go to Settings > Whitelist, add the user-agent
AEOifyBot. - Wordfence (WordPress) — Go to Firewall > Blocking, add a user-agent pattern allowlist for
AEOifyBot.
If your WAF supports IP-based allowlisting and you prefer that approach, contact us at support@aeoify.ai for our static IP address.
Verify it worked
After adding the whitelist rule:
- Go to your Dashboard and find the report for your site
- Click Rescan to run a fresh evaluation
- Compare the new scores against your previous report — if the crawler can now access your pages, you should see more detailed recommendations and potentially different scores