What a good AI crawler analytics tool should show beyond bot hit counts
Bot hit counts are the easy metric. Here is what we ask crawler analytics to show on client programs: verified bots, failures, page paths and links to citations.
When we start work with a new client, one of the first questions is whether AI crawlers can read the site at all. The usual first answer is a screenshot of a bot hit counter. It shows GPTBot visited 4,000 times last month. That number feels informative and almost never is.
Here is what we expect a good AI crawler analytics tool to show instead, and how we handle it day to day.
Start with whether the bot is real
Spoofed user agents are common, so a raw string count will credit AI engines with traffic that came from scrapers. We want logs where the request is verified by IP, with fakes filtered out. Promptwatch does this in Agent Analytics, which is a big part of why we trust the numbers enough to report them to clients.
We also want the bots separated by name. "AI crawler" is not one thing. The list we actually watch includes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and MistralAI-User. Different vendors use different agents for different jobs, so lumping them together hides the decision you actually need to make, such as which ones robots.txt should allow.
Pages, not totals
A total says nothing about coverage. What we want per page is simple: was it fetched, how recently, by which bot, and with what status. On a client with a large archive, the finding is often that the money pages are crawled and the useful supporting pages are not, or the reverse.
Stackable path filters matter more than people expect. Being able to isolate /pricing/ or a whole blog folder, then change the bot filter, turns a vague hunch into a list of URLs.
Errors, with an explanation
A failed fetch on a page that should be citable is the most actionable row in any report. Blocked by a firewall rule, server timeout, a redirect chain, a blanket robots.txt disallow: these have different fixes and different owners on the client side. Promptwatch lists error tracking with failure explanations, which saves us from guessing which of those it is. When we hand a client's developer a ticket, we want the evidence attached.
Crawl to citation, per page
This is where a log viewer ends and a visibility platform starts. For each page we want to see whether the crawl turned into a citation, and how often. Promptwatch shows a citation rate per page in its crawl-to-citation path. Two readings come out of it:
- Crawled often, rarely cited. The bot can read the page. The content is either not answering the prompt or someone else answers it better. This is a content job.
- Cited, rarely crawled. The engine may be leaning on an older copy. Check freshness and whether the page is being updated.
Neither reading is possible if the crawler tool is separate from the citation tool. We have tried stitching exports together in spreadsheets. It works once and then nobody maintains it.
Visits and conversions
The end of the chain is a person arriving from an AI answer. The 360 Insights view in Promptwatch joins crawler logs, citation clicks and AI-referred visits per page, and Topic Insights shows where pages underperform for their topic. For clients who care about revenue, the visitor analytics layer (script, GTM template or WordPress plugin) adds conversions on top. Sceptical finance teams respond better to "AI-referred visitors converted" than to "bots visited more."
What we do with the cheap options
Not every client needs a platform. If a site is already on Cloudflare, the free AI Crawl Control dashboard is a fine starting point for seeing request volume, status codes, paths and robots.txt violations, and for blocking agents. Known Agents is another lightweight route to bot analytics. Both stop at the crawl, though. Neither reports whether you were cited. We use them as a baseline and move clients to a full platform once the question becomes "so what, did it help?"
Plan limits to know about
Crawler logs are not on every Promptwatch plan. Essential ($95/mo) has none. They start on Professional at $245/mo with a 25M log allowance, and Business ($579/mo) raises it to 100M. Agency plans include crawler logs, with Kick-off at $199/mo. If you are buying mainly for crawler analytics, do not choose Essential and expect to find them.
Setting it up without drama
Logs reach Promptwatch through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint or the WordPress plugin. We ask for CDN access in week one, because log history only starts when the pipe is connected. That is the single most common avoidable delay.
What crawler data cannot do
It cannot prove a crawl caused a citation, and it cannot promise any AI engine will recommend a client. It shows access and behavior. That is still valuable, because "the bot cannot get in" is the cheapest problem to fix and the one most worth ruling out first.
Our checklist is short. Verified bots, named agents, page paths, explained errors, crawl-to-citation, visits. If a tool cannot show the last three, it is a counter, and we treat it like one.