The AI Crawler Names We Allow, Block, or Watch, and Why
A practitioner's guide to real AI bot names, what each is for, and the policy we start from for client sites before looking at the logs.
Crawler policy is where a surprising number of client programs go wrong. Either someone blocked everything in a panic about training data and lost search visibility with it, or nobody looked and the firewall did the blocking for them.
Our audit post covers the process. This one is a field guide to the names themselves and the default stance we take on each before we see a single log line.
Know the three jobs
AI bots fall into three broad jobs. Mixing them up is the root of most bad policy.
Training collection. Bots that gather content to train models. Blocking them affects whether future models learn from your pages. It does not directly affect whether a live search answer can cite you.
Search indexing. Bots that build the index an AI search product draws on. Blocking these removes you from that product's results.
User-triggered fetching. Bots that retrieve a page because a person asked a question that needed it. Blocking these means the engine cannot read your page at the moment someone asks about it.
A single company often runs one bot per job. That is why a blanket "block OpenAI" rule is blunt.
The names we see most in client logs
These are names that Promptwatch's Agent Analytics tracks, and they cover what we encounter most.
- GPTBot, OAI-SearchBot, ChatGPT-User: OpenAI's collection, search indexing, and user-triggered fetching. Note there is no bot called "ChatGPTBot", a mistake we see in client robots.txt files.
- ClaudeBot, Claude-User, Claude-SearchBot: Anthropic's equivalents across collection, user fetch, and search.
- PerplexityBot and Perplexity-User: indexing and user-triggered fetching for Perplexity.
- Google-Agent and GoogleOther: Google's agent and general-purpose crawlers, separate from the main search crawler.
- Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher: Meta's bots, relevant if Meta AI visibility matters to the client.
- MistralAI-User: Mistral's user-triggered fetcher.
We stop short of describing exactly what each bot does beyond its general role. Vendors change their documentation, and we point clients at each vendor's own page for specifics.
Our starting policy
We never apply a policy without discussing the client's business. But our default, for a company that wants to be found in AI answers, looks like this:
Allow search and user-triggered bots. If the goal is being cited, the bots that index and fetch for live answers need access. That generally means OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
Make a deliberate decision on training bots. Some clients are happy to be part of training data. Others, such as publishers with licensing concerns, are not. This is a business and legal choice, not a technical one. We write down who decided and when.
Watch the rest. We do not block unfamiliar bots by reflex. We check whether they are real, and we look at what they request.
Real versus spoofed
One catch: anyone can set a user agent string. A scraper can call itself GPTBot. Logs that only match on user agent will overstate AI crawler activity and can lead to the wrong policy. This is the reason we favour IP-verified data. Promptwatch's Agent Analytics filters out spoofed user agents by verifying IPs, which means the traffic we count as AI crawling is traffic that actually came from the named source.
Where policy lives, and where it breaks
Robots.txt is the first place, but not the only one.
- CDN or WAF rules often block bots by category regardless of robots.txt. We have seen a security rule introduced for unrelated reasons quietly cut off every AI crawler.
- Rate limiting can return errors to bots that hit a site frequently.
- JavaScript-only rendering may leave bots with an empty page.
- Login walls and geoblocks do the same.
For each, the log is the truth. A robots.txt that allows OAI-SearchBot means nothing if the CDN returns 403 to it.
How we confirm what is happening
- Connect crawler logs. Promptwatch integrates through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP, or the WordPress plugin. We pick whatever the client already uses.
- Look at which bots hit which paths, and what the response codes were. Agent Analytics shows error tracking with failure explanations.
- Compare with citations. The crawl-to-citation path shows a citation rate per page, so we can spot pages that bots hit often but never earn citations from.
- Fix access errors first, content second.
Crawler logs require the Professional plan or higher, or any agency plan. Essential does not include them, which is a point we raise before a client chooses a tier.
The robots.txt we do not write
We avoid copying someone's "block all AI bots" template. We also avoid pasting a long list of obscure user agents we cannot verify. A short, readable file that reflects a decision beats a long one that nobody understands. Annotate each section with a comment and a date.
Reviewing quarterly
Bot names change, and companies add new ones. We revisit the policy every quarter and after any infrastructure change. Promptwatch's AI Bots directory lists over 440 crawlers, which makes it easier to identify an unfamiliar name before deciding what to do about it.
Our recommendation
Decide policy by job, not by company. Allow the bots that index and fetch for live answers, make a conscious choice about training bots, and verify with IP-checked logs rather than user agents. We do that with Agent Analytics in Promptwatch. Start at promptwatch.com if you want logs you can trust.