How We Audit AI Crawler Access Before Any GEO Work
Our agency checklist for verifying robots rules, CDN controls, server responses, rendered content, and AI crawler logs before content optimization begins.
We audit crawler access before recommending new GEO content. If a search or live retrieval bot cannot fetch the relevant page, rewriting that page does not solve the immediate problem. The audit tells us whether the next ticket belongs with infrastructure, web development, content, or measurement.
Access is not a single yes or no setting. A site can allow a bot in robots.txt while a CDN blocks its request. A server can return successful HTML that contains little useful text until JavaScript runs. One provider's training crawler can be blocked while its search crawler remains open. We test the layers separately so the client receives a finding it can act on.
Define the pages and bot purposes
We start with a bounded URL set. It usually includes the homepage, commercial pages tied to the engagement, important documentation or editorial sources, and a few recently published URLs. A complete site crawl can follow, but a first audit needs pages with known business and prompt relevance.
Then we separate AI agents by purpose. Training crawlers gather material for model development. Search index crawlers discover pages for a provider's search experience. Live citation fetchers retrieve material in response to a user request. Promptwatch's crawlability documentation uses GPTBot and ClaudeBot as training examples, OAI-SearchBot, Claude-SearchBot, and PerplexityBot as search examples, and ChatGPT-User, Claude-User, and Perplexity-User as live fetch examples.
Those names are examples, not a permanent policy list. We check each provider's current official crawler documentation during the audit. Providers can change user agents, IP publication, controls, and stated purposes. The client's policy owner must also decide whether training access is acceptable. That decision is separate from whether the organization wants its public pages eligible for search and live citation.
Inspect robots.txt without stopping there
We retrieve robots.txt as an external user would and examine rules for the relevant agents. We look for broad wildcards, copied block lists, conflicting groups, and paths that unintentionally cover important content. We also verify that the file itself returns consistently on the relevant hostnames.
An allow rule does not prove access. Robots.txt expresses instructions to compliant crawlers; it does not override a firewall, authentication, geofence, rate limit, or broken route. A disallow rule can explain why a compliant bot stays away, but silence in a log is not enough to prove the crawler read and obeyed that rule.
Google-Extended needs careful wording. Promptwatch's guide describes it as a robots token used for certain Gemini training and grounding controls while Google continues crawling with its usual agents. We confirm the current meaning in Google's official documentation before advising a client because the business consequence is different from blocking ordinary Google Search crawling.
Test the edge and origin response
The next layer is the CDN, web application firewall, bot manager, and hosting edge. Security products often classify automated requests separately from browser traffic. We review rules and logs for blocks, challenges, rate limits, and rewritten responses. A normal browser session does not reproduce those conditions.
Where the client uses Cloudflare, the Promptwatch Cloudflare integration page documents two ingestion paths: Logpush for Cloudflare Enterprise and a Worker path for other plans. Other stacks need their own supported log route. We verify where traffic terminates before choosing the source. Logs from an origin can miss a request rejected at the edge.
For each test URL, we record hostname, path, bot purpose, observed status, response time, redirect destination, and response body notes. Redirects should reach the intended canonical page. Repeated errors need the exact rule or application fault attached where possible. We do not label every missing crawl as a block because a bot may simply not have requested the page during the selected period.
Check what the crawler can read
A successful status code is necessary but not sufficient. We inspect the returned HTML without relying on client-side rendering. The title, main copy, links, canonical information, and structured data relevant to the page should be present and coherent in the response a simple fetch receives.
If the useful content appears only after JavaScript executes, a browser-based audit may look healthy while a crawler gets an empty shell. We document that gap for the development team. We also check authentication walls, cookie gates, locale redirects, and consent behavior when they alter the response.
Promptwatch's AI crawler insights guide explains its raw log fields, status and path filters, and per-page views. It also notes that user-agent strings can be spoofed. Where a provider publishes IP ranges, verification adds confidence; agents without a published range require more caution. We never treat a user-agent match alone as proof of identity when stronger verification is available.
Compare access with discovery and citation
Once logs flow, we compare expected pages with actual requests. A sitemap or Search Console inventory helps reveal pages that should exist but receive no observed crawler traffic. Successful crawls followed by no citations shift the investigation toward content coverage and source selection. Citations with no recent observed crawl may rely on another retrieval path or previously available material, so we avoid declaring a simple causal sequence.
Promptwatch's vendor-published CDN and AI crawler study argues for reading edge logs beside citation and visitor data. We use that as a diagnostic model, not independent proof that crawl frequency causes citations.
Our audit deliverable is a matrix of tested pages and agents, evidence for each finding, the policy decision required, and a named owner. We rank access failures ahead of content polish. Training choices go to legal, security, and policy owners; search and citation blocks go to the team responsible for discoverability; rendering defects go to development.
We run this work through Promptwatch when the client's stack supports its crawler integrations because the log, citation, and page views sit in one project. The tool does not decide what a company should allow. It helps us show what requested each page, what response it received, and where the evidence chain stops.