How to check whether your website is technically ready for AI search engines and AI crawlers
Our technical readiness audit for AI search engines and AI crawlers: the robots.txt, firewall, rendering and log checks we run first on every client site.
Technical readiness is the first thing we audit on any new AI search engagement, before prompts, before content. The reason is simple: we have more than once spent weeks on content for a client whose pages the crawlers could not fetch. This is the order we now check things in.
1. Robots.txt, read per bot
Open the file and read it as each crawler would. The names that matter: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, plus Google's and Meta's fetchers in logs. Training crawlers and search crawlers are different decisions. A client may want to appear in ChatGPT's search answers and still opt out of training. Robots.txt can say that, but only if someone has written it deliberately.
We look for old template rules, blanket disallows and wildcard rules that catch more than intended. Promptwatch's free robots.txt generator and its AI Bots directory (440+ crawlers) are handy when a client asks what a bot name means.
2. The layers above robots.txt
This is where most of our findings come from. A firewall or bot-protection rule can return a 403 to an unfamiliar user agent before robots.txt is ever read. Sites often have this by accident, from a security tightening nobody connected to AI traffic. Check WAF settings, bot management, rate limits and CDN rules. If the client is on Cloudflare, AI Crawl Control is a free first look at which AI crawlers request pages and what status they get, although it only sees traffic that passes through Cloudflare.
3. What the bot actually receives
View source, not the rendered page. Some AI crawlers do not execute JavaScript like a browser, so a single-page app that builds its content client-side may look empty to them. We confirm that headlines, body text, prices and key facts are in the raw HTML. For most clients the fix is server-side rendering or static generation on the pages that matter.
4. Indexing and structured data
We check Google Search Console and Bing Webmaster Tools for indexing problems, then confirm schema markup matches what is visible, canonicals are sensible and the sitemap is current. Bing deserves attention because its traffic data includes Microsoft Copilot. IndexNow is a nice extra where the CMS supports it. We treat llms.txt as optional, and we never let a client believe it can stand in for crawlable pages.
5. Verify with logs
Configuration tells you what should happen. Logs tell you what did. This is where we use Promptwatch, because it connects the pieces.
- Agentic Readiness Check: a free first pass we use in sales conversations.
- Site audit: weekly Site Health runs with 21 checks, a health score, what changed since the last run, and a weekly email.
- Agent Analytics: IP-verified AI crawler logs, so spoofed user agents are filtered out. We see which bots fetched which pages, what status codes came back, why failures happened, and the crawl-to-citation path with a citation rate per page.
Logs come in through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP, or the WordPress plugin. Plan matters here: crawler logs start on Professional at $245 a month, and Essential at $95 does not include them. We tell clients that before they buy.
6. Close the loop with citations
Being crawled is not the same as being cited. A page that bots fetch daily but engines never cite is a content problem. A page bots never fetch is a technical one. Promptwatch's citation rate per page lets us tell the two apart quickly, and the Action Board turns what we find into prioritized tasks. TreeHouse, the company behind Pararius and Huurwoningen.nl, reported AI search reaching about 25% of its organic conversions after fixing crawler access and automating weekly reporting through the Promptwatch MCP. That result is theirs, and not a promise.
A one-page checklist you can copy
- Each relevant bot is allowed or blocked on purpose in robots.txt.
- No firewall or CDN rule returns 403 or 5xx to the search and user fetchers.
- Key content is present in the HTML source.
- Indexing is healthy in Search Console and Bing Webmaster Tools.
- Schema matches visible content; sitemap is current.
- Logs show the bots you want returning to the pages you want cited.
- You recheck weekly, because a deploy can break access without warning.
What passing means
A clean audit means your site is eligible to be cited. It does not mean an engine will recommend you, and no tool can promise that. If you want to run the logs part yourself, Promptwatch is where we would start, with the free readiness check before anything paid.