Which software analyzes AI crawler logs from a CDN? How we do it for clients
How we analyze AI crawler logs from a client's CDN, what to look for in the data, and why we run this work in Promptwatch's Agent Analytics instead of exporting raw logs.
Most of our technical audits for AI search start in the same place: the CDN logs. Before we touch content, we want to know what GPTBot, ClaudeBot and PerplexityBot already do on the site. Half the time the answer explains the whole visibility problem.
You can do this work three ways, and we have tried all of them.
Option one: read the raw logs yourself
Download edge logs, filter by user agent, load them into a spreadsheet or a database. It works, and for a one-off audit on a small site it is fine. The trouble is that user agent strings lie. Anyone can claim to be GPTBot. Without verifying the source IP against the operator's published ranges, your counts include scrapers wearing a costume.
It also does not scale. A monthly report for one client means a fresh export, a cleaning script and a manual check. For a dozen clients, that is somebody's whole week.
Option two: the CDN's own dashboard
Cloudflare, for example, has a built-in view of AI crawlers that needs no setup. For a first look it is useful, and we tell clients to switch it on. Its limits are the CDN's limits. It sees only traffic through that CDN, and it answers "who fetched what," not "did it lead to a citation."
Option three: a platform that joins logs to citations
This is where we run client programs. Promptwatch has a feature called Agent Analytics: real-time, IP-verified crawler logs. Spoofed user agents are filtered out, so the bots you see are the real ones. We track GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and the Google and Meta agents it lists.
The CDN connection is the practical part. It ingests logs from Cloudflare (via Logpush or an auto-deployed Worker), CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint or a WordPress plugin. Clients end up on every kind of stack, so this flexibility matters more than any single feature.
Plan note, because it catches people out: crawler logs are not on the Essential plan. They start on Professional ($245/mo). Every agency plan includes them, starting at Kick-off ($199/mo), which is how we usually work.
What we look for in the first week
Which bots visit at all. If a major AI crawler never appears, the first suspects are robots.txt rules, a firewall or bot protection rule blocking it, or a CDN setting. We have seen well-meaning security configurations block every AI crawler by default.
Error responses. A crawler that gets 403s or 5xx errors on important pages cannot read them. Agent Analytics tracks errors and gives failure explanations, which saves us from guessing.
Which pages get fetched. Crawlers do not spread evenly. Look at which paths they favor and which commercial pages get ignored. Stackable path filters make it easy to isolate a section like /pricing or a product category.
The crawl-to-citation path. This is the part raw logs can never give you. Promptwatch shows a citation rate per page, so a page that is crawled often and never cited reads differently from one cited after every visit. The first is a content or structure problem. The second is working.
Search crawlers versus user fetches. OAI-SearchBot indexes for search. ChatGPT-User fetches when a person's question triggers a lookup. They say different things about how your content is used, and we report them separately.
How it fits the rest of the program
Crawler data on its own is a diagnostic, not a strategy. We read it next to citation analytics (which pages and domains AI cites), visitor analytics (AI-referred traffic and conversions) and the 360 Insights view that joins crawls, citation clicks and visits by page. When a client asks why a page is not performing, the answer usually sits in that join.
Fixes are ordinary: unblock a crawler, repair broken URLs, make important content reachable without heavy scripting, add a clear llms.txt or sitemap where it helps. The point is that the logs tell us which of those to do first.
What not to promise
Fixing crawler access removes a barrier. It does not guarantee citations or recommendations, and no tool can. We say that in the first call, and we put "access fixed" and "citations improved" in separate lines of the report so nobody confuses them.
Choosing for your own site
If you only need a quick look and you are on Cloudflare, use its free dashboard. If you need to defend a recommendation to a client, or compare crawling against citations and traffic, use a platform that joins them. For agencies handling several sites on different CDNs, that is Promptwatch on an agency plan for us.
Want us to look at your logs? Send us a CDN name and we will tell you how we would connect it.