Agentic Web Optimization: A Brand Guide
How we run an agentic web audit for a client: AI agent access by purpose, fact consistency across pages and feeds, Merchant Center and ACP readiness, protocol triage, and a measurement baseline.
When a client asks us to make their site "ready for AI agents," we start with an audit, not a build. The phrase covers very different visitors. Some agents index pages for an AI search product. Some collect training data. Some fetch a single page because a person asked an assistant about it. Others read product feeds so a shopping answer can show a price. Each one needs something different from the site, and a few of them need nothing from the marketing team at all.
This is how we run that audit, in the order we run it.
What we ask for on day one
Before we look at a single page, we ask the client for five things: read access to the CDN or hosting logs, the current robots.txt and firewall rules, Merchant Center access if they sell products, an export of any product feed, and a short list of the pages that matter commercially. We also ask one question that has nothing to do with SEO: who in the company decides whether AI companies may train on the site's content?
That last answer often takes longest. It belongs with legal or whoever handles content licensing, and we don't make that call for a client. We make sure it gets made on purpose.
Step one: map agents by purpose
We build an access matrix with one row per agent and one column per priority page. The rows come from each vendor's own documentation, and we re-check it on every audit because names and behaviors change.
For search, OpenAI's OAI-SearchBot surfaces sites in ChatGPT search, Anthropic's Claude-SearchBot indexes pages for Claude's search quality, and Perplexity's PerplexityBot surfaces and links sites in Perplexity search. Perplexity states PerplexityBot isn't used for training. When we find one of these blocked by a copied "block all AI" list, it's usually the first fix we recommend, because the block costs the client an answer surface and protects nothing.
For training, the agents are GPTBot and ClaudeBot. We report their current status and leave the decision with the policy owner. Google-Extended gets a separate line in the report. Google documents it as a robots.txt control token for Gemini training and grounding, with no user agent of its own and no effect on Google Search inclusion. Clients tend to read it as a training switch. Because grounding is in scope too, we spell out what they'd give up before anyone adds it.
For user-triggered fetches, the agents are ChatGPT-User, Claude-User and Perplexity-User. This is where robots.txt stops being the whole story. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Claude-User follows robots.txt, like every Anthropic bot, and Anthropic notes each bot has to be blocked separately on every subdomain. If a client wants to stop a user-triggered agent, it takes a firewall rule, and we make sure they understand who it hurts: their own customers asking an assistant to read a page.
We wrote up the access part of this process in more detail in how we audit AI crawler access before any GEO work.
Step two: read the logs, not the rules
A robots.txt file says what the client asked for. Logs say what happened. We connect Promptwatch's Agent Analytics to the client's CDN (it supports Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN and custom HTTP) and look at which AI crawlers request which pages, which requests fail, and the crawl-to-citation path from a fetch to a cited answer.
We don't trust a user agent string on its own, since anyone can fake one. Where the vendor publishes IP lists, as OpenAI and Perplexity do for their agents, we check requests against them. Google points to its published ranges and reverse DNS. Perplexity recommends combining user agent and IP matching in firewall rules, and that's how we write the rules we hand over. Anthropic warns IP blocking may not work reliably for its bots, so for those we rely on robots.txt.
Step three: a fact consistency pass
Agents compare. We take each priority product or plan and record how its name, price and main terms appear on the page, in the feed, in Merchant Center and on any partner listing. Mismatches go in a table with the owner who can fix each one.
We also check the boring pages that agents and shopping programs actually depend on. Is the return policy on its own crawlable URL, with the window and return costs stated in text? Is there a support page with a URL, email or phone number? Are prices in text rather than in an image or a PDF? Do headings say what's under them? None of this needs a source to justify it. It's what a careful reader would want, and agents read carefully.
Step four: feeds and protocol readiness
For Google, we check the Merchant Center account is in good standing with approved products, that a return policy is defined (return cost, return window, link to the full policy), and that customer support info includes at least one of a website, email or phone. Google's UCP-powered checkout requires all of that. That checkout puts a Buy button on eligible listings in AI Mode and Gemini while the client stays merchant of record. We're plain with clients about its reach: it's "available for select merchants at this time," through an early access form, for products eligible in the US, Canada and Australia, and many categories are excluded, including subscriptions, pre-orders, used goods and services.
For ChatGPT, we look at whether the client should send a product feed through ACP, which OpenAI extended to product discovery on 24 March 2026. Shopify merchants are covered through Shopify Catalog. OpenAI's merchant FAQ says a feed isn't strictly required if ChatGPT already crawls the site, but it gives more control. When a client does build one, we check that every product has a stable id and at least one variant with its own id and title, per the ACP product spec. OpenAI has also said it's moving away from a standalone Instant Checkout, so purchases complete on the merchant's own site.
AP2 and MCP get a short paragraph each in the report. AP2 is about proving a user authorized an agent's purchase; v0.2 has been donated to the FIDO Alliance. We list it as something to watch. MCP connects AI apps to tools and data, and it isn't a checkout protocol. We say that explicitly because vendors blur the two.
Step five: a measurement baseline
We set the baseline before any fix ships, otherwise there's nothing to compare against. We run this on Promptwatch, the platform we run client programs on. Agent Analytics gives us crawler activity and errors per page. Citation analytics shows which client pages get cited and which Reddit threads, YouTube videos and third-party domains are cited instead. Visitor analytics, installed with a lightweight script or the GTM template, counts AI-referred visits and conversions; ChatGPT's own utm_source=chatgpt.com tag helps, but it covers one assistant. For product clients we add ChatGPT Shopping tracking.
What the client gets
The deliverable is the access matrix, the fact mismatch table, a feed and protocol readiness note, and the baseline. Every finding has an owner: platform team for access, ecommerce for feeds, support for policy pages, legal for training decisions, marketing for content.
Content gaps come last. Where the citation data shows prompts that competitors win and the client doesn't, we use Promptwatch's Content Agents to draft the missing pages. Drafts land in a review inbox, and someone on our team reads each one before it's published to Webflow, Framer or WordPress.
If you want this audit run on your site, write to hello@1001seomedia.com.