Why Spoofed User Agents Wreck AI Crawler Reports
Why user-agent-only crawler counts mislead, how we spot fake AI bot traffic, and why we rely on IP-verified logs in client reporting.
A client once forwarded us a chart from their hosting dashboard. It showed a large spike in "GPTBot" traffic and a note from their developer: "AI is crawling us like crazy, this must be good." We asked for the raw logs. A good part of that traffic came from addresses that had nothing to do with the company that operates the bot.
This happens more than people expect. It is also a reporting problem we can solve, if we are careful.
How user agents work, and why they cannot be trusted
A user agent is a string a client sends to say who it is. Browsers send one, and so do crawlers. There is no authentication. Any script can declare itself to be any bot.
That is fine for casual traffic analysis. It is a problem for decisions. If a scraper calls itself GPTBot, then a count of GPTBot requests includes it. Allow rules written against that string will let it in. Block rules will block the genuine bot and the scraper alike, but only if the genuine bot's string matches.
Why scrapers pretend
Some sites treat known AI bots more generously than unknown ones. Scrapers know that. Calling yourself a well-known crawler is a cheap way to slip past basic filtering. Others do it for no reason beyond sloppy testing. The motive does not matter much. The result is that the labelled traffic is a mix of the real thing and imitations.
What goes wrong in practice
Inflated activity. Reports say bot X visited ten thousand times. Part of that is noise. A team celebrates, or a CFO asks why bandwidth costs rose.
Wrong block decisions. A security team sees suspicious volume under an AI bot name and blocks the string. Now the real bot is cut off too, and citations quietly dry up.
Misread crawl-to-citation links. If the crawl numbers are padded, the ratio between crawls and citations is meaningless. A page may look heavily crawled and never cited when in reality it was barely fetched by the real bot.
False comfort. A page shows plenty of "AI crawler" hits, so nobody checks whether the real one reached it.
How verification works
Several AI companies document how to confirm their bots, and the vendor's own page is where to check. Verification means checking that the request's source address matches what the company says belongs to its crawler. A request that claims to be a bot but comes from elsewhere is not that bot.
You can do this by hand with reverse lookups and published lists, and some teams do. It is tedious, it goes out of date, and it is easy to get wrong across a dozen bots.
Promptwatch's Agent Analytics handles it in the product. It logs AI crawler activity in real time, verifies IPs, and filters out spoofed user agents. The tracked bots include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, and MistralAI-User. When we hand a client a chart, the counts mean what they say.
How we spot trouble in raw logs anyway
Even with verified data, we keep a few habits for raw logs.
- Look at request patterns. A real indexing bot tends to fetch a broad range of pages at a sane rate. A scraper often hits the same thing repeatedly or fetches only a specific subset.
- Check response codes by bot. A spoofer that is blocked at the edge may show a different pattern to the genuine bot.
- Compare timing. Bursts at unusual intervals deserve a closer look.
- Look at the paths. A bot requesting login or admin paths is not behaving like an indexing crawler.
What to tell a client
We keep it short. "User-agent counts can be faked. We report verified crawler activity only, so the numbers are conservative." Clients appreciate conservative. It protects them from reporting a gain that disappears when someone investigates.
We also warn them before they set up alerts on user-agent volume, because those can fire on junk.
Connecting the logs
Logs come from the infrastructure, so setup depends on the stack. Promptwatch supports Cloudflare via Logpush or an auto-deployed Worker, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin. For clients on Cloudflare we usually take the Worker route for speed. For larger setups we plan the log pipeline with their infrastructure team first.
Crawler logs are on Professional and above and on agency plans. Essential has none, so a client who buys that tier for the logs will be disappointed. We say that before purchase.
What good reporting looks like
A crawler section in our monthly report includes:
- Verified activity by bot, with a note on which bots matter to the client's goals.
- Errors by page with the explanation, sorted by business importance.
- Pages that are crawled but not cited, and pages cited but rarely crawled.
- Actions taken and actions pending.
We leave out vanity counts. The number of bot hits tells you little. The number of important pages that the right bot reached successfully tells you a lot.
A word on cost
Cleaning up unverified logs by hand costs analyst time, and that time has a price. We have found it cheaper to use verified data from the start than to reconcile suspect counts later.
Our recommendation
Do not report AI crawler activity from user-agent strings alone. Use IP-verified logs and treat everything else as unconfirmed. We rely on Agent Analytics in Promptwatch for this because verification is built in and the crawl-to-citation path sits beside it. You can see how it works at promptwatch.com.