How do I audit my website's visibility in AI search engines? Our onboarding audit
Our AI search visibility audit starts with crawler access, then indexing, content and prompt testing. This is the sequence we run in the first weeks of a client program.
Every client program we run begins with an audit of the site's visibility in AI search engines. We have changed the order of it over time, and the biggest lesson was to check access before content. Here is the sequence we use now.
Why we start with access
New clients tend to hand us a list of articles they want "optimized." Sometimes the articles are fine. The problem is that the bots which feed AI answers cannot read them. We have found it more useful to prove access first, since it is faster to fix than content and it gates everything else.
We check robots.txt for the real bot names: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-User and Claude-SearchBot from Anthropic, PerplexityBot and Perplexity-User, plus Google and Meta agents. OpenAI separates its crawlers by purpose, with OAI-SearchBot connected to ChatGPT search and GPTBot to training, so we discuss with the client which they want to allow instead of copying a template.
Then we look past robots.txt. CDN rules, bot-protection settings and WAF configurations can block AI crawlers while the robots file looks permissive. Only server-side evidence settles the question.
How we get that evidence
We use Agent Analytics in Promptwatch. It shows real-time AI crawler logs, IP-verified so that spoofed user agents are filtered out, with a record of which pages were fetched, which requests failed, and an explanation of the failure. It also reports a citation rate per page, which connects crawling to citing. We can say "this bot hit this page, got a 403, and this page has not been cited since."
Scoping note: crawler logs begin on the Professional plan ($245 a month), not Essential ($95). When a client is on a tight budget, we say that at the start. Logs need to be connected through an integration such as Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint or the WordPress plugin, so we also check early which of those the client can set up.
Step two: indexing
AI engines that search the web depend on search indexes. We verify Google Search Console and Bing Webmaster Tools for every client, since Copilot draws on Bing. We look for noindex tags, canonical errors, soft 404s, and pages whose main content is rendered only by client-side JavaScript, because some crawlers do not run scripts.
Promptwatch's weekly Site Health audit helps here. It runs 21 checks, produces a health score, reports what changed since the last run, and emails a summary. We use it as the standing technical baseline for the whole engagement.
Step three: content and structure
Next we review the pages that should be cited. What we look for:
- A direct answer high on the page.
- Headings that mirror real questions.
- Specific facts stated plainly: pricing, specifications, locations, dates.
- Structured data that is valid and matches what the page shows.
- Evidence of who wrote the page and when it was last updated.
We do not rewrite every page. We pick the pages that map to the client's highest-value prompts.
Step four: test with prompts
Together with the client we write 30 to 50 prompts: category questions, comparisons, "best X for Y," plus a few that include the brand name. We tag them by topic and intent. Promptwatch runs them daily per model, country and language, so our baseline is a pattern, not a single screenshot. We record Visibility Score, Share of Voice and Position in Answer, and we keep the set frozen for the first month.
Step five: citations and offsite presence
Finally we look at what engines cite. Citation analytics show which pages and domains appear for each prompt, including Reddit and YouTube. The offsite mentions list shows where the brand is named on third-party pages without a link. Often the gap is not on the client's own site at all but in the places the engines already trust.
The output is a list of prompts where competitors are cited and the client is not. It becomes the content plan, and the Action Board helps us prioritize it.
What the client gets
We deliver the audit as a short document with three sections: access fixes (do now), content fixes (do next), and offsite work (ongoing). Each finding comes with the evidence behind it. We do not include predictions about how much visibility will improve, since we cannot promise that.
What we tell clients it cannot do
An audit removes obstacles and gives you a baseline. It does not guarantee an engine will name your brand. Answers vary between runs and change with model updates, and each platform weights sources differently. We re-run the audit monthly with the same prompts and the same pages, and report what moved.
If you are auditing your own site
Begin with robots.txt and your firewall. Verify Bing and Google webmaster accounts. Write your prompt set. Then decide whether a platform is worth it. If you can get crawler logs and citations in the same view, the audit becomes something you repeat instead of something you do once.