1001 SEO Media
All posts
By 1001 SEO Mediatechnical SEOAEO auditAI crawlerscrawler logs

AEO technical audit tools: what we check first

The order we run an AEO technical audit in: robots.txt and AI crawler access, verified crawler logs, crawl-to-citation rates, site health and IndexNow.

When a client says AI engines never mention them, the cause is sometimes content and sometimes authority. But before we spend a day on either, we check whether the engines can read the site at all. A blocked crawler, an erroring template or a page missing from every link path will undo any amount of content work, and those problems are cheap to find if you look in the right order.

This is the order we run a technical AEO audit in, with the tools we use at each step. Most of it runs on Promptwatch, the platform we run client programs on, plus a couple of free checks anyone can use. For the wider tool picture, see our guide to top AEO tools.

Step 1: is anything blocked on purpose?

We start with the rules the site publishes. A robots.txt written years ago for Googlebot can block AI crawlers by accident, through a wildcard rule or a default group nobody revisited. A CDN or firewall rule can block them too, which robots.txt won't show.

Promptwatch's free tools cover the published side, and none of them needs an account. The Agentic Readiness Check reads a site's robots.txt the way nine different AI crawlers would. It also looks for Content Signals, OAuth discovery, A2A and MCP cards, and an llms.txt file. For one page at a time, there's the AI Crawler URL Inspector. And the AI robots.txt generator writes a separate group for each crawler you name.

Promptwatch free tools page listing the ChatGPT Query Fan-Out Generator, AI Brand Visibility Report, Agentic Readiness Check, AI SEO Audit, LLMs.txt generator, AI Crawler URL Inspector and other free GEO tools The free tools index on Promptwatch's site, as it looked on November 14, 2026.

If you edit robots.txt by hand, one detail from Promptwatch's November 6, 2026 changelog matters. When ExaSearchBot has no group of its own, it obeys whatever you wrote for Googlebot and Bingbot, not the default group. The generator adds a group for it any time search-engine rules are included. Writing the file yourself? Decide on Exa explicitly.

We don't treat llms.txt as a ranking factor. We check it because it's cheap, and because a broken or stale one is a sign that nobody owns the site's AI-facing files.

Step 2: are the crawlers actually coming?

Published rules tell you what should happen. Logs tell you what does. Audits often skip this step, because the logs aren't in a form anyone can read.

Promptwatch's Agent Analytics records AI crawler visits from server or CDN logs. Each visit is checked against the IP ranges its operator publishes, which removes anything that only fakes a crawler's user agent. OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User are covered; so are Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User, Google-Agent and GoogleOther, the three Meta crawlers, and MistralAI-User. Exa's two crawlers are logged as well, but never marked Verified, since Exa doesn't publish IP ranges.

Promptwatch Crawler Logs page with the AI crawlers filter open over a stacked bar chart of daily visits, listing GPT Bot, SearchBot, ChatGPT Citations and Ads Bot from OpenAI and Claude Bot, Claude Citations and Claude Search Bot from Anthropic, most with a Verified badge, and a toggle to also include unconfirmed visits Crawler Logs with the verified AI crawler filter open, as shown in Promptwatch's changelog. The project is example data.

The logs can come from the WordPress plugin, a custom HTTP endpoint, Cloudflare (Logpush, or a Worker that Promptwatch deploys), AWS CloudFront, Akamai DataStream 2, Fastly, Google Cloud CDN, or a Vercel or Netlify log drain. Since November 13, 2026, HubSpot sites can send logs too, by putting their own Cloudflare zone in front of HubSpot. You need Professional ($245 a month) or any agency plan for crawler logs.

What we look for in the first week:

  • A provider that is absent. Promptwatch's AI crawler traffic report put OpenAI at 79.8% of verified AI crawler requests for August 31 to September 6, 2026. In the June 8 to 14 week it had been 94.8%. If a site's logs show no OpenAI crawlers at all, something is blocking them.
  • Errors. Error tracking explains failures per bot and URL. A 403 for one crawler and a 200 for another usually means a firewall rule.
  • The Host column, added November 13, 2026, which warns about logs from hostnames other than the project website. Staging domains and old hostnames show up here.

New crawlers appear fast, so we don't audit against a fixed list. Promptwatch's Meta-WebIndexer report tracked that one crawler's share of AI crawler requests rising from about 2.2% in mid-July 2026 to 37.8% on August 9, 2026. That report identifies crawlers by user agent, not IP. A rule written in spring could have blocked it without anyone noticing.

Promptwatch daily bar chart of Meta-WebIndexer's share of AI crawler requests from July 11 to August 9, 2026, with a first spike around July 20 and a second ramp from August 5 Daily Meta-WebIndexer share, July 11 to August 9, 2026, from Promptwatch's Meta-WebIndexer data report. Captured on November 14, 2026.

Step 3: crawled, but cited?

Once crawlers are reaching the site, the next question is whether crawled pages get cited. Agent Analytics follows each page from crawl to citation and gives it a citation rate. A page crawled often but never cited is a content problem: the engine reads it and finds nothing worth quoting. If a page ought to be cited and no crawler ever fetches it, that's access or discovery.

That split decides who gets the ticket. Content problems go to writers. Access problems go to developers. Without it, both teams guess.

On November 4, 2026, Promptwatch changed how its GEO 360 view classifies bots. Index-building crawlers like OAI-SearchBot, Claude-SearchBot and Meta-WebIndexer now count as AI crawls. Only a bot that fetches a page in the moment, to answer someone's question, counts as a citation crawl, and ChatGPT-User is the example Promptwatch gives. Comparing totals with an older export, you'll see citation crawls lower and AI crawls higher. Nothing changed on the site.

Step 4: site health, weekly

Every week the site audit runs 21 checks and produces a health score, a list of what changed since the previous run, and an email on Tuesday. Page allowances run from 1,500 pages on Essential to 25,000 on Business, and 10,000 to 75,000 on the agency plans. Site Content looks at page length, titles and descriptions, headings, duplicate pages and freshness. Site Linking finds broken links, orphaned pages, and pages the sitemap leaves out.

Promptwatch site health Linking view with an internal linking map grouped by URL path, marking orphan pages, pages with one or two inlinks, and redirects or errors The site health Linking view in Promptwatch, shown in its changelog with an example project.

The internal linking map, grouped by URL path, is the view we use most in AEO work. It colors orphan pages, pages with only one or two inlinks, and redirects or errors in different shades. An orphan page is one crawlers can only find through the sitemap, if at all. The Pages list records where each URL came from (sitemap, link, Search Console, or the site map crawl), which tells you how it was found.

Ignore rules (November 13, 2026) let you skip a page, a folder, or a regex pattern from the next audit. We wrote about how we use them in what we exclude from client site audits. The short version: we check citations and crawler activity before excluding anything.

Step 5: tell the engines what changed

After a fix ships, we don't wait for crawlers to rediscover it. IndexNow, added to Promptwatch on November 13, 2026, submits changed pages to search engines that support the protocol. You add a key in the project's settings, either generated or one your site already serves, and verify it. Then Site > IndexNow lists pages to submit, with filters for pages that changed since their last submission, pages never submitted, and failures. Each attempt keeps IndexNow's reason if it failed.

The technical SEO platforms, from our catalog

Enterprise technical SEO platforms have moved toward AI search. Botify now describes serving content to agents and Markdown delivery, sold through a demo with no published price. Lumar's AI search visibility is a custom-priced add-on next to its crawl diagnostics. Cloudflare AI Crawl Control comes free with any Cloudflare plan. It lists the AI crawlers requesting pages and lets you allow or block each one, but on the Free plan detection relies on user-agent strings, and it runs no prompts and reports no citations.

Each covers part of the audit. Promptwatch puts verified crawler visits and citations for the same URL side by side, with a citation rate per page, and that join is what tells an access problem from a content problem. Promptwatch's TreeHouse case study, for Pararius and Huurwoningen.nl, lists fixing crawler access among the changes behind their AI search growth.

Where to start

Put your domain into Promptwatch's free AI Visibility Report. The scan covers ChatGPT, Gemini and Perplexity and includes a crawler access check, which takes care of step 1 in a few minutes. A blocked crawler in that report goes to the top of the list, ahead of everything else here.