How to Detect AI Agents Using or Citing My Content
How we set up AI crawler logs, cited-URL tracking and AI referral analytics for a new client in week one, and why fetches, citations and clicks stay in separate columns.
Most clients arrive with some evidence already. Someone spotted GPTBot in a server report, or a handful of sessions from chatgpt.com turned up in analytics, and the question becomes "are AI tools using our content?" Each of those observations is real. Each one also proves less than it seems to.
So the first week of every engagement goes on separating three things that people tend to merge: AI agents fetching the site, AI answers citing it, and humans clicking through from those answers. Here is how we set that up, day by day.
Before day one: agree on what each signal can prove
We send the client a short note before the kickoff call, because the reporting arguments we want to avoid start here.
| Signal | Where we look | What it proves | Blind spot |
|---|---|---|---|
| Agent fetch | CDN or server logs, by user agent, checked against IP lists | A named bot requested the page and got a response | Whether any answer used or cited it |
| Citation | Answers to the prompts we track, with cited URLs recorded | An engine used the page as a source for that prompt | Prompts outside our set; whether anyone clicked |
| Click-through | Analytics referrers, utm_source=chatgpt.com | A person arrived from an AI answer | Citations that got no click; the prompt behind the visit |
The two sentences we repeat most often in week one: a crawl is not a citation, and a citation without a click leaves no trace in analytics. Analytics on its own undercounts how often a site is cited, because a citation nobody clicks produces no visit. Logs read as if they were citations overstate it.
Day one: find where the traffic terminates
Before connecting anything, we work out where requests actually land. If the site sits behind a CDN, logs from the origin server miss every request the edge blocked or challenged, and those are often the ones we most need to see.
Then we connect the client's CDN to Agent Analytics in Promptwatch, the platform we run client programs on. The supported routes are Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and a custom HTTP endpoint for stacks that don't fit those. The slow part is rarely the integration itself. It's finding the person who owns the CDN account and getting them on a call.
Day two: sort bots by purpose and check who they really are
Once logs flow, we group requests by what each bot is for, using the vendors' own documentation. OpenAI's bot page separates GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (visits a user triggered from ChatGPT or a Custom GPT). Anthropic's crawler article does the same for ClaudeBot, Claude-SearchBot and Claude-User. Perplexity's bot guide covers PerplexityBot, which it says isn't used for training, and Perplexity-User for fetches a user requested.
Training hits go in their own column. They matter for the client's content policy, but they say nothing about AI search visibility. The search crawlers tell us whether pages can be found. The user-triggered fetchers are the closest thing in a log to a live answer, since someone's conversation pulled the page, but even those don't show whether the answer went on to cite it.
Then we check identity. User agents can be spoofed, and Google's crawler documentation says to verify through published IP ranges or reverse DNS. OpenAI publishes lists at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json. Perplexity publishes perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. Anthropic warns that IP blocking may not work reliably for its bots. Requests that claim a bot name but fail verification get reported separately, never folded into the totals.
Two corrections we make often. Google-Extended has no user agent, so nobody will find it in a log. It's a robots.txt token for Gemini training and grounding, and it doesn't affect Google Search. And GoogleOther, which does appear in logs, is a generic crawler for Google product teams rather than an AI-specific bot. We also flag that the user-triggered fetchers play by different rules: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. For the deeper access checks, our AI crawler access audit covers robots rules, edge responses and rendering.
Day three: build the prompt set and take a citation baseline
No log will ever tell us that ChatGPT cited a page. We have to ask. We draft the first prompt set from the client's sales questions and category searches, load it into Promptwatch, and let it run across the engines the client cares about.
What we read from the baseline is citation analytics at page level first, then domain level. Page level tells us which URLs are doing the work. Domain level shows which third parties sit next to the client in answers. We also pull the Reddit and YouTube citations, because a thread or video about the client's category can be cited where the client's own pages are not, and that changes what the content plan should be.
Day four: install visitor analytics
For the click side we install Promptwatch visitor analytics, normally through the GTM template since the client's marketing team usually owns the container. A lightweight script works when there's no tag manager. We agree on which conversions count with the client rather than guessing, and we confirm that utm_source=chatgpt.com visits, which OpenAI's publisher FAQ says ChatGPT adds to referral links, are arriving with the parameter intact. A redirect that drops query strings would remove it, so we click through a test link on the main landing pages.
Day five: the first joined readout
By Friday we can put the three signals side by side for the client's priority URLs. The crawl-to-citation path in Agent Analytics shows which fetched pages went on to be cited, and error tracking shows which pages bots tried to read and failed. We sort each page into one of a few states. Not fetched at all means we fix access before anything else. Fetched but not cited points to content or relevance. Cited but no visits means the client is visible without traffic, which isn't a problem to solve on its own. Cited with converting visits is the page to protect.
We are careful about what the readout claims. A crawl followed by a citation is a sequence we can observe, not proof that the crawl caused the citation.
What this costs to run
For a single brand we start on Professional at $245 a month, which is the plan that lists a crawler-log allowance (25M logs) alongside 150 prompts and 1M visitor events. Clients with several brands go on an agency plan, where every tier includes crawler logs and unlimited projects. Week one ends with three connected data sources and a baseline. The work of acting on it starts in week two. If you want this set up for your site, email hello@1001seomedia.com.