AI Crawler Visibility Tool for Bot Activity on Your Website: 'AI Crawlers' in Cloudflare, Ahrefs, Semrush (2026)
Cloudflare AI Crawl Control, Ahrefs Bot Analytics and Semrush's AI crawler reports compared from an agency seat, plus the question none of them answer.
Most of our clients already pay for at least one of Cloudflare, Ahrefs or Semrush. All three now have a screen with "AI crawlers" or "AI bots" on it. When a client asks which one to use, the answer depends on what they want to learn, because these screens look at different data and answer different questions.
Here is how we use each one, what it gets right, and the gap we fill elsewhere.
Cloudflare AI Crawl Control: the control panel
Cloudflare renamed AI Audit to AI Crawl Control, and every plan has it. Because it lives at the edge, it counts whatever reaches a proxied hostname. When we open it on a client zone, we're reading which AI companies are knocking (OpenAI, Anthropic, Google, Microsoft, Meta, ByteDance), which of their bots got through and which bounced, who ignored robots.txt, and which URLs and status codes were involved. The same screen is where you flip a crawler to allowed or blocked. Charging bots per crawl exists too, though only as a private beta.
How much you see depends on what the client pays Cloudflare. A Free zone identifies bots from the user-agent string alone and only keeps a day of metrics, and referral counts aren't shown at all below the paid tiers. Enterprise customers who also buy Bot Management get detection IDs and can pick their own date range. Engineers who want the raw numbers can query them through the GraphQL Analytics API.
For us it's the enforcement layer. If a WAF rule is quietly throwing 403s at ClaudeBot, this is where we spot it first.
Ahrefs Bot Analytics: server logs for SEO teams
Ahrefs costs nothing extra for this right now: Bot Analytics is in a free beta and reads your server logs rather than guessing. Bots get bucketed into a dozen groups, with AI assistants sitting next to search engines and SEO crawlers, and you can click into a single bot such as GPTBot or Googlebot to see which pages it hit and what status codes came back. One switch filters the view down to AI bots only.
Getting logs in is mostly a CDN question. Cloudflare sites can use Logpush if they're on Enterprise, or a Worker on any tier. Vercel, Fastly and Amazon setups have their own connectors. The catch for us is that Bot Analytics is its own product, apart from Brand Radar and from Ahrefs Web Analytics. The crawl and the AI mentions end up in two different screens, and you do the matching yourself.
If the SEO team already works in Ahrefs, it's a reasonable free place to start looking at real bot traffic.
Semrush: three features with similar names
Semrush has more than one thing that sounds like AI crawler analytics, and they do different jobs.
Site Audit is a simulation. Semrush crawls your site itself and grades it for AI search with a health score, flags AI-related issues, and has a widget that reads your robots.txt and tells you which of eight big AI crawlers you're shutting out. You can even switch the audit to crawl as OpenAI-Search. Useful, but it describes what a bot would run into, not what a bot actually did.
Log File Analyzer is the real-traffic option. Feed it your server logs and narrow them to OpenAI's three agents (GPTBot, OAI-SearchBot, ChatGPT-User) or to ClaudeBot.
Semrush Enterprise goes further, at a price it doesn't publish. Site Intelligence tracks around 30 bots, the AI Optimization product has its own "Agent Analytics" screen, and Crawler Profiles let you replay a crawl the way ChatGPT's agents or Googlebot would. Clients sometimes ask about the name: Promptwatch also calls its crawler product Agent Analytics, and its founders say theirs came first in the category. Different companies, different products.
How they compare
| Cloudflare AI Crawl Control | Ahrefs Bot Analytics | Semrush | Promptwatch Agent Analytics | |
|---|---|---|---|---|
| Data source | Edge requests on proxied hosts | Your logs via CDN connectors | Simulated audit crawl, uploaded logs, Enterprise bot data | Your logs via CDN connectors |
| Published price | All plans; more on paid tiers | Free during beta | Enterprise bot features have no public price | From Professional $245/mo, Kick-off $199/mo for agencies |
| Links a crawl to a prompt and citation | No | No | Not documented | Yes, crawl-to-citation path |
| Links to AI-referred visits | No | No (separate product) | Not documented | Yes, via visitor analytics |
The question none of the three answer
Every client eventually asks the same thing: "The bot came. Did it matter?" Crawler data alone can't say. A fetch only counts if the page later shows up in an answer, for a prompt the client cares about, and ideally sends someone who converts.
That's the reason we answer it in Promptwatch, the platform we run client programs on. Its Agent Analytics streams crawler hits live (ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther and Meta's AI crawler) and flags errors. The difference from the three tools above is where those logs live: in the same project as the tracked prompts and their citations. So "fetched on Tuesday, cited on Friday for this prompt" is something we read off a screen instead of building in a spreadsheet. The visitor analytics script then shows whether those cited pages sent AI-referred sessions that converted.
Setup runs through whatever CDN the client already has. On Cloudflare that means Logpush for Enterprise zones or a Worker Promptwatch deploys for everyone else, as long as the hostname is proxied (orange cloud). CloudFront, Fastly, Vercel, Netlify, Akamai and Google Cloud CDN are supported, and anything else can post logs over HTTP. On the brand side, crawler logs arrive with Professional at $245/mo (25M) and grow to 100M on Business at $579/mo. Agency plans carry them from the $199 Kick-off tier (10M) up through Growth at $399 (25M) and Scale at $799 (100M). Essential lists no crawler-log allowance, so we don't sell this work on it.
What we actually tell clients
Keep Cloudflare AI Crawl Control on. It's where you allow and block, and it costs nothing extra. If your SEO team is in Ahrefs or Semrush, their crawler screens are fine for spotting blocked bots and broken status codes, and the Semrush Site Audit check is a quick robots.txt sanity test.
None of that replaces the join. When the question is which crawls turned into citations and which citations turned into revenue, we read it in Promptwatch. Our crawl-to-citation write-up walks through how we read that view on a weekly call.
Want us to set it up? Email hello@1001seomedia.com with your CDN and whether you're on Cloudflare Enterprise.