How to check if AI bots can crawl my website: the audit we run for clients
The audit we run to check whether AI bots can crawl a client's website: robots.txt, firewall and CDN rules, a user-agent test, and logs that prove what actually arrived.
Why this is our first check
On almost every new GEO engagement we find at least one access problem. A client has good content and weak AI visibility, and the cause is not the content. It is a bot rule nobody remembers adding. So before we plan prompts or write anything, we check whether the bots can read the site.
Here is the audit, in the order we run it.
Which bots we test
We test by name, since rules are written per user agent: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI. ClaudeBot, Claude-User and Claude-SearchBot from Anthropic. PerplexityBot and Perplexity-User. Google-Agent and GoogleOther. Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalFetcher. MistralAI-User.
One correction we make often: use the exact names. A rule written for an invented bot name does nothing, and it is usually a sign the rule was copied from a poorly sourced list.
We also separate purposes. Some bots collect training data, some index for search, and some fetch a page when a user asks. A client may reasonably block one purpose and allow another. That is a business decision, and we ask for it in writing.
Layer one: robots.txt
We read the file at the root of the domain and look for three things.
- Rules naming specific AI bots.
- A wildcard group with
Disallow: /, which can block anything without a more specific group. - Leftovers: a staging-site rule pushed to production, or a broad rule added years ago to stop a scraper.
We also read each operator's documentation for how user-triggered fetchers treat robots.txt, because they are not always handled like background crawlers. Google's Search Central documentation on how robots.txt groups are matched and tested is a good reference for the logic.
Layer two: firewall and CDN
This is where the surprises live. A web application firewall or bot-management setting can block by user agent, IP range, request rate or a general bot challenge, regardless of robots.txt. A client can have a perfectly open robots.txt and a firewall returning 403 to every AI crawler.
For Cloudflare clients, we start with AI Crawl Control, which is available on all plans and shows which AI crawlers request pages, grouped by operator, with status codes, popular paths and robots.txt violations. On Free it identifies crawlers by user agent only, and referral metrics need a paid plan.
Layer three: a direct request
A simple terminal test:
curl -I -A "OAI-SearchBot" example.com/some-page
We read the status code. 200 is a pass for that user agent. 403, 429 or a challenge page means something blocks it. We repeat for each bot.
This is a screen, not proof. It tests user-agent rules only. Real crawlers come from published IP ranges, and a firewall may treat them differently from our request. We say that in the audit report so nobody treats a green curl as the final word.
Layer four: logs
Logs prove what bots experienced. We look at requests per bot, status codes and paths, and we verify IPs, since user-agent strings can be faked.
We run this in Promptwatch's Agent Analytics. It records real-time, IP-verified AI crawler logs, filters out spoofed user agents, explains failures, and supports stackable path filters and CSV export. Logs can come from Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP source or the WordPress plugin.
The crawl-to-citation path is what we value most. It gives a citation rate per page, which separates pages that are fetched and cited from pages that are fetched and ignored. That second group is where content work pays off. The first group is where we stop worrying.
Plan note: crawler logs start on Professional at $245 a month. Essential does not include them. Every agency plan does.
Free starting points
Promptwatch also offers free tools that help with the first pass: an Agentic Readiness Check, an AI Bots directory of more than 440 crawlers, and robots.txt and llms.txt generators. We use the directory when a client's logs show an unfamiliar bot.
How we write it up
Each finding becomes a row: bot, layer where it was blocked, evidence, proposed change, owner. After the fix, we re-check logs a few days later to confirm that the bot now gets a 200.
Typical fixes: remove a stale Disallow, add an explicit allow group for chosen bots, adjust a firewall rule, or whitelist verified IP ranges.
What we do not claim
Access is a precondition, not an outcome. A bot that can crawl a page may still never cite it. And we do not promise a citation after a fix. We promise a clearer picture of what the bots saw.
If you want to run the same audit with logs that verify IPs, start with Promptwatch.