1001 SEO Media
All posts
By 1001 SEO MediaGEOAI searchtechnical SEO

OpenAI Search Crawler OAI-SearchBot Documentation

Official OpenAI docs: OAI-SearchBot is for ChatGPT Search. GPTBot is training. ChatGPT-User is a user fetch. We log crawls in Promptwatch Agent Analytics.

OpenAI search crawler OAI-SearchBot documentation lives on Overview of OpenAI Crawlers. This is a technical SEO ticket, not a mention-dashboard ticket. OAI-SearchBot is for search. It surfaces websites in ChatGPT's search features. Sites opted out will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from published IP ranges (searchbot.json).

Example user-agent (version may change): compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Robots.txt fetches may include an extra robots.txt marker in the UA.

We put this in the audit before anyone argues about GEO copy. If the search bot gets a 403, the mention program is theater.

Do not mix the bots

GPTBot is training data. ChatGPT-User is used when a user (or Custom GPT) fetches a page. Robots.txt may not apply to ChatGPT-User the same way. It is not the Search opt-out control. Use OAI-SearchBot in robots.txt to manage Search.

The publishers FAQ repeats: block OAI-SearchBot and you hurt summaries and snippets. utm_source=chatgpt.com tags clicks when they happen. Referrals are a different post.

CDN WAF presets labeled "block AI bots" often catch all three. We read the access log for the actual UA, then we change the preset. Robots.txt intent is not ground truth.

Google-Extended is a Google control. It does not govern OAI-SearchBot. Do not copy a Google robots line into the OpenAI ticket.

After the bot is allowed

Allowing the bot does not guarantee a citation. It removes a self-inflicted 403. Passage work is still how to get cited by ChatGPT. Google Overviews stay on Search Central: our reading.

Promptwatch Agent Analytics is where we log whether AI crawlers fetched the URL and whether that path later showed up as a citation. Paid plans also store the ChatGPT Search answer on prompts you typed. Essential is $95/mo. Explore is ChatGPT prompts only, not the crawl stack we brief. Kick-off is $199/mo with unlimited projects and 10 seats. Professional is $245/mo. Business is $579/mo. Visitor analytics (script or GTM) attach the click if one arrives.

Otterly, Peec, and Profound do not replace the robots.txt change. Ahrefs and Semrush stay for the rest of the crawl graph.

What we will not do

We will not block GPTBot and tell a client they opted out of ChatGPT Search. We will not promise citations from a robots allow. We will not skip log verification.

FAQ

Is OAI-SearchBot the same as GPTBot?

No. Search versus training.

Can we block GPTBot and allow OAI-SearchBot?

Yes. Those are different decisions. Document both in the ticket.

Does Promptwatch fetch as OAI-SearchBot?

Promptwatch monitors the UI and logs crawler hits you already received. It is not a replacement for allowing OpenAI's bot.

What to do this week

  1. Read the bots page.
  2. Allow OAI-SearchBot on pages you want in ChatGPT Search.
  3. Allow the published IP ranges at the CDN. Confirm in logs.
  4. Connect Agent Analytics in Promptwatch.
  5. Load the prompts those pages should win, then email hello@1001seomedia.com if you want the robots and CDN pass done on the account.

Related articles