robots.txt and AI Crawlers: What to Allow and What to Block
A practical framework for setting AI crawler policy by purpose while accounting for provider changes, CDN enforcement, and citation goals.
There is no universal robots.txt policy for AI crawlers. A publisher protecting licensed archives, a software company seeking citations for public documentation, and a private portal holding customer data have different requirements. Copying a block list from a forum ignores those differences and can disable the search access a GEO program needs.
We treat robots policy as a business and technical decision. The business decides which uses of public content it accepts. The technical team translates that choice into current user-agent rules and edge controls. The GEO team verifies the result in logs rather than assuming the file worked.
Classify the crawler by purpose
The broad label AI bot hides several jobs. A training bot collects content for model development. A search index bot gathers pages for an AI search product. A user-triggered or citation fetcher requests a page while serving an answer. Blocking one purpose should not be treated as a decision about all three.
Promptwatch's crawlability guide gives current examples. It places GPTBot and ClaudeBot in the training category, OAI-SearchBot, Claude-SearchBot, and PerplexityBot in search indexing, then ChatGPT-User, Claude-User, and Perplexity-User in live retrieval. The guide also explains that Google-Extended is a control token with a different role from Google's ordinary search crawlers.
We use that page to understand Promptwatch's classifications, but we do not paste its list into robots.txt without verification. Before every policy change, we open the current official documentation for OpenAI, Anthropic, Perplexity, Google, and any other provider in scope. User-agent names and described uses can change. Some providers publish IP information, while others may not. A dated policy register should link to the documentation consulted.
Decide what the organization wants
The first policy question is whether public content may be used for training. Legal, editorial, licensing, security, and product teams may all have a view. GEO performance does not settle that choice. A company can decline training access while still evaluating whether it wants search indexing and live retrieval.
The second question concerns visibility. If a brand wants eligible public pages to appear as sources in an AI search experience, blocking that provider's search and citation agents works against the stated goal. This does not mean every path should be open. Public help articles may be suitable while account areas, staging hosts, internal search results, and private files remain inaccessible.
The third question is operational risk. A crawler that causes unacceptable load may need rate or infrastructure handling. Robots rules are not a substitute for capacity planning or access control. Sensitive material should require authentication and proper authorization; it should never depend on a polite crawler directive for protection.
We turn these decisions into a matrix with one row per provider and purpose. Each row records the intended policy, paths covered, business owner, official documentation date, implementation location, and verification result. That record matters when a future team asks why one agent is allowed and another is not.
Understand what robots.txt cannot do
Robots.txt tells compliant agents what they may fetch. It does not force a crawler to comply, verify the crawler's identity, or secure a private URL. It also cannot make a blocked request pass through a CDN firewall.
A site may contain an explicit allow rule while its bot-management product returns a 403 or challenge. A rate limiter may admit the first request and reject later ones. A route can return a 404 after a migration even though robots.txt permits it. The inverse is possible too: edge logs can show repeated requests for robots.txt while the crawler avoids disallowed pages.
That is why we pair the file review with request evidence. Promptwatch's AI crawler insights documentation describes filters for crawler, path, date, and status, along with raw request fields. It also explains the limits of user-agent identity and the use of provider IP ranges where available. A claimed bot name without verification is weaker evidence than a request that matches a provider's published range.
Apply narrow rules and test them
We prefer explicit, purpose-led groups over a single broad directive copied across every AI-related name. The implementation should match the approved matrix and existing site conventions. We review wildcards carefully because one short path pattern can cover more URLs than its author intended.
After deployment, we fetch robots.txt from each relevant hostname and environment. We confirm the served version, caching behavior, content type, and whether redirects land on the expected file. Then we watch logs for requests to robots.txt and subsequent page activity. A configuration in source control is not enough if the CDN still serves an older cached copy.
CDN settings receive a separate test. For Cloudflare users, Promptwatch documents its Cloudflare crawler-log connection, including an Enterprise Logpush route and a Worker route for other plans. The relevant point for policy is that edge-level evidence can expose requests that never reach the application. Other CDN providers require the equivalent supported log path.
We use representative public pages and paths that should remain blocked. The test record includes expected behavior and observed status for each crawler purpose. We avoid impersonation claims: setting a user-agent in a local request can test a rule branch, but it does not reproduce a verified provider request or prove that the provider will crawl the page.
Review policy as the products change
Robots policy needs a review date. A provider may split one agent into separate training and search agents, revise controls, or publish new verification details. A newly launched site section may also inherit a rule written for a different path.
The operating review compares the policy register with current official provider documentation and recent edge logs. It asks whether intended search and live agents can reach priority pages, whether blocked purposes remain blocked, and whether errors come from robots, CDN rules, or the application. Changes receive an owner and a fresh observation period.
We use Promptwatch on client programs to observe crawler requests and connect access findings with citations. We do not outsource policy to the platform. The approved matrix remains the source of intent, provider documentation supplies the current agent definitions, and logs show whether the deployed controls match both.