AI Crawler Logs in Agent Analytics: Crawl-to-Citation Path and Error Tracking

If ChatGPTBot never fetched the page, the H1 is not the ticket. We open Promptwatch Agent Analytics first: crawler logs, errors, then crawl-to-citation. Content work waits.

This is the same order we use in how to get cited by ChatGPT. Robots and CDN blocks are still the cheap misses.

What the logs actually show

Agent Analytics is a live view of AI agents on the site. The user agents we care about on most accounts: ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, Google-Agent, Meta. OpenAI also documents OAI-SearchBot as a separate bot from GPTBot. Those are different allow rules.

Crawl discovery lists pages found, indexed, and blocked. Top pages show what gets hit most. Path filters (exact or partial) keep a blog and an app on the same hostname from mixing. Filters persist between log screens, which matters when we are in a client call and flipping views.

Error tracking is the first column we sort. A 403 or a challenge page is an engineering ticket. We do not rewrite the passage until the fetch succeeds. CSV export is how we hand the same rows to a developer who will not live in the GEO login.

Logs come through the CDN the site already uses: Cloudflare, CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or a custom HTTP endpoint. WordPress behind Cloudflare can already stream logs even though WordPress CMS publish is still listed as coming soon.

Crawl-to-citation is the join

Crawl-to-citation connects a fetched URL to whether that URL later appears as a citation, plus a citation rate. Plenty of pages get read and never used. That is a content or authority problem, not a robots problem. Plenty of pages never get read. That is the opposite ticket.

Sitemap intelligence supports more than one sitemap per project and surfaces page-level prompt opportunities. We upload the sitemap we already trust. We do not invent a second sitemap for "AI."

Professional includes 25M crawler logs at $245/mo. Kick-off includes 10M at $199/mo (unlimited projects, 10 seats). Essential at $95/mo is the first brand program; crawler volume grows as you move up. Business is $579/mo and 100M logs. Explore is free ChatGPT prompts only. It is not this stack. G2: 4.7/5.

Otterly at $29 does not give you crawl-to-citation. Peec at $95 is mentions. Profound Starter at $99 annual is ChatGPT mentions. We keep Screaming Frog for HTML. We keep GSC for Google. Agent Analytics is the AI bot log.

Paid engines for the answer side stay the usual set: ChatGPT, Gemini, Claude, Perplexity, Grok, Llama, DeepSeek, Mistral, Copilot, AI Overviews, AI Mode. The crawler list and the answer list are not the same thing. We do not blur them on the slide.

What we will not blame on content

We will not invoice "the model cannot see this page" until the log shows a fetch or a hard block. We will not treat a missing citation after a successful crawl as a CDN outage. We will not claim instant alerts on errors. Daily paid checks, then we read the log.

FAQ

Can engineering dump CDN logs into a sheet instead?

They can. They will not get citation rate next to the status code without glue. We would rather not glue that on a retainer.

Do we need a new robots.txt for every bot?

We need the bots in scope allowed, and we need the log to prove it. A public bots directory exists on promptwatch.com if a new user agent shows up. We do not add mystery agents to robots because a blog post named them.

How we run it

Connect the CDN you already pay for. Confirm ChatGPTBot, ClaudeBot, PerplexityBot, and GoogleOther are not blocked.
Filter to the two URLs we plan to cite. Read errors first.
Check crawl-to-citation on those URLs after a quiet week of prompts.
File engineering tickets for blocks. File content tickets only for pages that were fetched and unused.
hello@1001seomedia.com if you want us to wire the logs. Bring the CDN name and the current robots.txt.