Getting AI crawler data from clients on Shopify, Webflow, and WordPress
Many clients cannot hand over server logs. Here is how we get AI crawler visibility on hosted platforms, and what each setup needs.
The kickoff call goes the same way every time. We ask for access to server logs. The client's marketing manager says they are on Shopify, or Webflow, or a managed WordPress host, and that nobody has ever seen a log file. That is true. On hosted platforms the web server belongs to someone else, and the logs are not part of the package.
For years that meant giving up on crawler analysis for a large share of small and mid-size clients. It no longer has to. This post covers what we ask for, in which order, and what to tell the client who is nervous about touching DNS.
Why we want crawler logs at all
AI answers cannot cite what a retrieval system could not fetch. Without logs, an access problem is invisible. A 403 from a bot protection rule, a block in robots.txt that nobody meant to set, or a crawler that simply never arrives all look the same from the outside: the client is not being cited, and nobody knows why.
Logs turn that into something specific. We can see which bots requested which pages, which got errors, and which expected bots never showed up. We also learn to separate training crawlers from search index crawlers and from live fetchers that run when a user asks a question. Blocking one does not mean blocking the others, and the real names matter: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and so on. Our note on the crawler names we allow and why covers the policy side.
What to try first, in order
Check what the client can reach. Before any setup, find out who controls DNS and whether the site sits behind a CDN such as Cloudflare. That one answer decides the route.
If the site is on WordPress, use the plugin route. WordPress needs no CDN change at all.
If the site is on Shopify, Webflow, WP Engine, or Kinsta, the route is almost always a CDN in front of the site, with logs pushed from there.
If no CDN exists, the realistic options are putting one in front (which is a bigger change) or settling for indirect evidence such as fetch tests, robots.txt review, and referral traffic.
Be honest with the client about that last case. Without logs we can infer, but we cannot observe, and we say so in the report.
The setup behind the scenes
On the platform we use, the changelog gives two dated facts that cover most clients. The first: according to the Promptwatch changelog, an official WordPress plugin arrived on August 18, 2026, and it brings content publishing, visitor analytics, and crawler logging together in one install, with no CDN setup or code changes required. The visitor analytics and publishing sides apply to any organisation, and the crawler log feature follows the plan limits described below.
The second: as of October 6 the same changelog lists Cloudflare Logpush as the default crawler log setup for WP Engine, Webflow, Shopify, and Kinsta sites. Those hosts do not give you raw server logs, so routing through Cloudflare is how you get the request data.
The client work for the Cloudflare route is small but needs a person with the right access. They need to add the domain to Cloudflare, or confirm it is already there, and then either enable Logpush or deploy the Worker that Promptwatch sets up. We walk through the steps in our guide to connecting Cloudflare crawler logs to Promptwatch. We send the client a one-page checklist and book a 30-minute screen share rather than trade emails for a week.
A note on verification
Spoofed user agents wreck crawler reports. Anyone can send a request that claims to be GPTBot. Counting those as real inflates the numbers and hides actual access problems. Promptwatch's agent analytics verifies crawlers by IP, and an October changelog entry says verified data now uses historical IP ranges from each provider, so crawler visits from ranges a provider has since retired are counted again instead of being dropped. For us that means fewer unexplained gaps in the historical data we show a client. We unpack the problem in why spoofed user agents wreck AI crawler reports.
Why we use Promptwatch for this
Promptwatch is the platform we run client programs on, and the main reason is that crawler logs sit next to citations and visitor analytics in one project. A client with a hosted site gets the same crawl-to-citation view as one with their own servers, once the log feed exists. See promptwatch.com for the integrations list.
Which plan does it need
Plan limits shape the scope conversation, so we state them up front. On the brand plans, crawler logs start on Professional at $245 per month, and Essential has none. Business adds a much higher log volume. Every agency plan includes crawler logs, with 10M monthly logs on Kick-off ($199 per month), 25M on Growth ($399), and 100M on Scale ($799). Prices are monthly billing, and the free trial is on monthly billing only. If a client is on a plan with no crawler logs, we tell them the data does not exist, and suggest the upgrade only if the access question is likely to be the blocker.
What to tell the nervous client
Clients worry about three things. Whether their site will slow down, whether they are handing over customer data, and whether it can be undone. We answer with facts we know and avoid guessing about ones we do not. The log data is request metadata, such as URL, user agent, and status. It can be removed by disabling the log push. Where the client's compliance team needs detail, we refer them to Promptwatch's security documentation instead of paraphrasing, since Promptwatch states ISO/IEC 27001 certification and GDPR compliance in its own materials.
What we report once the data arrives
The first report is plain. Which bots visited, which pages they reached, which returned errors, and which expected bots were absent. Then the fixes: unblock a rule, repair a redirect chain, or correct a robots.txt line. We re-check after a few weeks and report the change.
A client who "cannot see logs" can usually see them within a week once someone owns the DNS question. That one conversation changes what we can diagnose for the rest of the engagement.