Best platform to monitor site health for AI search: what we use with clients
How we check site health for AI search on client programs: crawler access, audits, and the one platform that ties them to citations and AI-referred traffic.
When a client asks why they are missing from AI answers, we do not start with content. We start with the plumbing. In our experience a surprising share of visibility problems turn out to be access problems: a bot blocked at the CDN, a template that errors only for non-browser user agents, a section of the site that was never linked from anywhere crawlable.
Here is how we monitor site health for AI search, and which platform we run it on.
Why classic site health is not enough
A normal technical audit asks whether Googlebot can crawl and index a page. AI search adds a second population of visitors: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and a growing list of others. Each one can be allowed or blocked separately, and each one behaves differently. Some crawl for training, some fetch pages live when a user asks a question.
A robots.txt file written five years ago may block half of them by accident. A bot-protection rule may challenge them. A rate limit may throttle them. None of that shows in a ranking report.
What we check, in order
1. Can the bots reach the pages? We look at real server-side logs, not guesses. Promptwatch's Agent Analytics gives us IP-verified AI crawler logs, which matters because user agent strings are trivial to spoof. If a request claims to be GPTBot but comes from the wrong network, it gets filtered out. That keeps our picture honest.
2. What happens when they arrive? The same view shows errors and failure explanations. A 403 for ClaudeBot on every product page is a very different problem from a slow response on one archive URL.
3. Are fetched pages becoming cited pages? The crawl-to-citation view gives a citation rate per page. Pages that are fetched often but never cited usually have a content or structure problem. Pages that are cited but rarely fetched may be relying on cached material and are worth refreshing.
4. Does the rest of the site hold up? The weekly Site Health audit runs 21 checks, produces a health score and shows what changed since the last run. The Site Content and Site Linking reports help us spot thin pages and orphaned sections.
5. Does any of it reach revenue? Visitor analytics counts AI-referred visits and conversions, so we can tell the client whether fixing a crawl issue changed anything that matters to them.
Why we run this on Promptwatch
We have tried to stitch this together from separate tools: a log analyzer here, a crawler there, a mention tracker somewhere else. It works, but the joins are manual and the client never sees one coherent story. Promptwatch puts the audit, the crawler logs, the citation data and the visitor analytics in one workspace, and 360 Insights joins crawler logs, citation clicks and AI-referred visits per page. Search Insights also overlays Google Search Console and Bing Webmaster data, which helps because Bing feeds Copilot.
The Action Board then turns what the data shows into a prioritized to-do list. We still review every suggestion before it goes to a client, but it saves the first hour of every monthly review.
Cost depends on what you need. Essential at $95 a month includes the site audit, but crawler logs start on Professional at $245 a month. For agencies, every self-serve plan includes crawler logs, starting at $199 a month for Kick-off. Check the current pricing page before committing, as plans change.
A monthly routine that works
- Week one: read the audit, note new failures, and check crawler error rates by bot.
- Week two: fix the highest-impact access and linking problems with the client's developers.
- Week three: compare crawl and citation rates on the pages we touched.
- Week four: report on AI-referred visits and plan the next round.
It is dull work, and it is where a lot of the real wins come from.
What monitoring will not do
It will not guarantee a citation. Allowing the bots in is necessary, but whether an engine names your brand depends on the content, the sources around it and the question being asked. We tell clients that plainly. What good monitoring does is remove the excuses: if the bots reach the pages, the pages load cleanly and the content is strong, then the remaining work is editorial and off-site.
Questions to ask before you pick a platform
Whichever tool you choose, ask these:
- Does it show real crawler requests, or only check whether robots.txt looks permissive?
- Does it verify the source of crawler traffic?
- Can it connect crawls to citations on the same page?
- Does it show AI-referred visitors and conversions, or stop at mentions?
- Does the audit run on a schedule and tell you what changed?
Most prompt trackers answer none of these. That is not a criticism of what they were built for. It does mean they are the wrong tool for this particular job, and it is why our clients' site health work sits in Promptwatch.