How We Compare Client Crawler Logs to the Open-Web Mix
Promptwatch publishes weekly AI crawler shares across the open web. Here is how we hold a client's own logs against that mix to find blocked or unreachable providers.
OpenAI accounted for 79.8% of verified AI crawler requests in the week of August 31 to September 6, 2026. In the week of June 8 to June 14 it was 94.8%. The figures come from Promptwatch's AI crawler traffic report, which was published on September 9, 2026. The page now updates weekly, so we quote the week above and tell readers to check the live version for anything newer.
That mix is useful to us for a reason that has nothing to do with market share. It gives us a baseline to hold a client's own server logs against. This post describes how we do that.
What the report measures
The report counts weekly requests from five providers: OpenAI, Anthropic, Google, Perplexity and Mistral. Only requests whose IP address matches the operator's published ranges are included, so spoofed user agents are excluded. Weeks run Monday to Sunday, UTC. The counts come from raw server and CDN logs connected to Promptwatch's crawler analytics.
Two details change how the chart should be read. The shares are per week and always total 100%. A provider whose segment grows has a larger slice, which is not the same as making more requests. If OpenAI's share fell from 94.8% to 79.8%, other providers' shares rose, but the report's share view does not tell us the absolute counts. Second, provider totals hide the split between training crawlers and answer-time fetchers. The chart can be grouped by crawler to separate them.
What it does not tell us
This is the mix across the sites connected to one vendor's analytics. It is not a census of the internet, and it does not describe a given client's audience. A site's crawl mix depends on its topic, its size, its age and its access rules. A client whose mix differs from the report is not necessarily wrong. The comparison is a prompt to ask a question, not a verdict.
The report also says nothing about citations. A crawler visit is a fetch. Whether the fetched page gets used in an answer is a separate matter, and one we measure separately.
How we use it, step by step
The report's own advice is to check robots.txt, CDN and WAF rules for every provider, compare your logs to the mix, and look per crawler at whether OpenAI traffic is training, answer-time fetching, or both. Our process follows that.
- Get the client's raw logs, or at least a week of CDN data. Many clients do not know who owns the CDN configuration. Finding that person is half the work.
- Group requests by provider using verified IPs, not user agents. A spoofed GPTBot is common enough that user-agent counts alone mislead.
- Set the client's mix beside the report's. We do not expect a match. We look for gaps. The report's wording is that a provider large in the report but near zero in your logs suggests you are blocked or unreachable for it.
- Investigate each gap. Look at robots.txt first, then WAF and bot-management rules, then rate limiting, then whether the pages render without JavaScript. A bot that gets a 403 is a firewall problem. A bot that gets a 200 with an empty body is a rendering problem.
- Split by crawler. If OpenAI traffic is almost all training collection and little answer-time fetching, that tells us something different from the opposite pattern. The names matter: GPTBot, OAI-SearchBot and ChatGPT-User do different jobs. There is no bot called ChatGPTBot.
- Write down the policy decision. Whether to allow training crawlers is a business choice. We record who made it and when.
A check anyone can do
Even without a platform, a client can pull a week of access logs, filter for the bot names above, and count status codes by bot. If one provider is missing entirely, test it by requesting a page with that bot's user agent from outside the network and compare the response to a normal browser visit. This is manual and gets stale quickly, and it cannot verify IPs, but it will catch an obvious block.
The measurement step
Doing this every week by hand does not hold up. For client programs we use Agent Analytics in Promptwatch, the platform we run client programs on. It shows IP-verified AI crawler logs in real time and tracks the named bots, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, the Meta bots and MistralAI-User. Error tracking explains failed requests, path filters narrow the view, and CSV export lets us line the data up against the report's weekly mix.
The crawl-to-citation path adds the next question. It gives a citation rate per page, so we can see which pages bots fetch and which of those show up in tracked answers. A page that is crawled often and never cited is a content question. A page that is never crawled is an access question. We keep the two apart on purpose.
Setup runs through the CDN or host the client already has: Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin. Crawler logs start on the Professional plan at $245 a month, and they are on every agency plan. Essential has none, which is worth knowing before choosing a tier for a client whose problem is access.
What we tell clients
The open-web mix is a reference point and nothing more. The first thing we want from the comparison is a list of providers that are absent from the client's logs and shouldn't be. Then we fix access, wait a few weeks, and read the logs again. We do not promise that a fetch becomes a citation. We promise that the client will know whether the door is open.