1001 SEO Media
All posts
By 1001 SEO MediaAI crawlersTechnical SEOrobots.txtGooglebotGEO

Which AI crawler user agents should you allow or block on your website? How we handle it in a technical SEO audit

How our SEO and GEO team decides which AI crawler user agents to allow or block, why Googlebot already covers AI Overviews, and how we check it in the logs.

When a client asks us which AI crawlers to block, the conversation usually starts in the wrong place. They've read that "AI bots are scraping everything" and want a list to paste into robots.txt. We'd rather start with a different question: which AI answers do you want to appear in? The crawler rules follow from that.

We treat AI crawler access as one more section of the technical SEO audit, next to indexation, canonicals and internal links. It uses the same file, the same CDN settings and often the same mistakes.

Start with the crawler you already manage

For Google, there is no separate AI crawler to allow. Google Search Central says your Googlebot rules apply to Google Search and all its features, which includes AI Overviews and AI Mode. To be a supporting link in those, a page has to be indexed and eligible for a snippet. Google says there are no extra technical requirements beyond that.

So if your SEO is healthy, your Google AI visibility has the access it needs. If a staging rule or an overzealous WAF is blocking Googlebot, you're losing both classic rankings and AI features at once.

Google's list of common crawlers showing the Googlebot user agents and the products their robots.txt rules affect

Google's "List of Google's common crawlers" page, captured November 21, 2026.

The token that confuses people is Google-Extended. It has no user agent of its own. It tells Google whether crawled content may train future Gemini models and ground answers in Gemini Apps and Vertex AI. Google says it has no effect on Search inclusion and isn't a ranking signal. The name invites the assumption that blocking it keeps a site out of AI Overviews. It doesn't, and it was never meant to.

Our default policy for a commercial site

This is what we recommend unless the client has a licensing reason to do otherwise:

  • Allow OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot and Meta-WebIndexer. Each operator says these feed its AI search, and blocking them removes or reduces your pages there.
  • Leave alone the user-triggered fetchers: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Google-Agent. They fetch a page because a person asked a question. OpenAI, Perplexity, Meta and Google each say robots.txt may not govern them, so a disallow line mostly gives a false sense of control.
  • Decide on purpose for GPTBot, ClaudeBot, Meta-ExternalAgent and Google-Extended. These are about training. OpenAI and Google state that the training setting is independent of search, so a client can block training without losing citations from those two.

Then we write it down. A robots.txt decision that nobody documented gets reversed by the next developer who copies a "block all AI" snippet from a forum.

The parts robots.txt can't fix

Two technical issues sit outside the file entirely.

The first is the CDN or firewall. OpenAI tells site owners to allow its published IP ranges as well as the robots.txt token, and Perplexity says to combine user-agent rules with IP verification. A bot-protection setting that challenges unknown traffic can quietly block a crawler that robots.txt allows.

The second is subdomains. Anthropic asks for the opt-out on every subdomain. The same applies in reverse: a blog on a separate subdomain with its own robots.txt can block AI search crawlers the main site welcomes.

How we verify it for clients

We run client programs on Promptwatch, and crawler logs are the part we lean on in a technical audit. Its Agent Analytics reads the CDN or server logs (Cloudflare, CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or the WordPress plugin) and checks each request against the operator's published IP ranges, so spoofed user agents don't skew the picture. We look at which bots arrive, which pages they request, and which ones get errors.

The classic SEO link is Search Insights. It puts Google Search Console and Bing Webmaster clicks and impressions next to AI crawls, citations and AI clicks for the same pages. When a page is crawled by OAI-SearchBot, ranks in Google and still never gets cited, that's a content problem, not an access problem, and it goes on the content roadmap instead of the tech ticket list.

Crawler logs start on the Professional plan. If you want to see what that looks like on your own site before deciding on a robots.txt policy, book a Promptwatch demo and ask them to walk through Agent Analytics with your logs connected.