Technical Optimization for AI Crawlers: Schema, Crawlability, and Robots.txt

Technical GEO is still crawlability. Schema is a helper. A new text file is not a ranking system. We use Promptwatch technical recommendations next to Agent Analytics, then we change robots, CDN rules, and structured data the way we already would for Googlebot.

Google's own AI features and AI optimization notes still say: useful pages, crawlable, indexed, structured data that matches visible text. They do not say Google reads llms.txt to build AI Overviews. We will not claim that.

Crawlability before schema

Promptwatch crawlability analysis flags whether AI systems can fetch and parse the page. We pair that with Agent Analytics errors. A block in robots.txt or a bot fight on the CDN shows up as a failed fetch. That is the first ticket. See how to get cited by ChatGPT for the bot list we check by hand: GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, GoogleOther.

A free robots.txt generator and a bots directory live on promptwatch.com if we need a first draft. The live test is still the log. We do not ship a generated robots.txt that allows a bot the security team has not approved.

Schema suggestions in the product are enhancement ideas, not a new vocabulary. We mark up what the user can see: Product, FAQ, HowTo, Organization, LocalBusiness when those are true. We do not add FAQ schema for questions that are not on the page. Invisible FAQ is how you get a manual action and still fail the assistant.

llms.txt can exist. Promptwatch even has a generator. We will publish one if the client wants a machine-readable summary. We will not tell them Google Overviews will move because the file landed. Promptwatch's own research page says llms.txt has no proven impact on AI search. We treat it as optional documentation, like a nicer about page for bots.

What we change in practice

Allow the crawlers in scope. Confirm in logs.
Fix rendering so the claim we want cited is in the HTML, not only in a client-side widget.
Add or fix schema that restates visible facts.
Then rewrite the passage.

Otterly at $29 will not do this pass. Peec at $95 is mentions. Profound Starter at $99 annual is ChatGPT only. We still run Screaming Frog and Search Console. Promptwatch is the AI-crawler half plus the answer log on ChatGPT, Gemini, Claude, Perplexity, Grok, Llama, DeepSeek, Mistral, Copilot, AI Overviews, and AI Mode.

Essential is $95/mo and includes technical optimization on the plan. Professional is $245/mo. Business is $579/mo. Kick-off is $199/mo (unlimited projects and prompts, 10 seats). Explore is free, 10 ChatGPT prompts. G2: 4.7/5. Daily paid refresh. No instant alerts. No $29 Promptwatch plan.

What we will not sell as technical GEO

A paid "AI schema pack" from a local SEO vendor we have not verified. Invented BrightLocal or GeoRanker AI SKUs. A promise that llms.txt is required. A robots.txt that blocks GPTBot "for training" and then a complaint that ChatGPT Search cannot cite the docs.

Local businesses still need the local SEO playbook. GBP and NAP are not replaced by a Promptwatch schema suggestion.

FAQ

Should we block GPTBot and allow OAI-SearchBot only?

That is a legal and security choice. If Search is in scope, OAI-SearchBot needs to fetch. If you block both, we will not run a citation program and then blame the model.

Does schema get us into AI Overviews?

Google says structured data should match the page. It does not say a new schema type is an Overviews lever. We add accurate markup. We measure Overviews in the same Promptwatch project, not in the schema validator.

How we run it

Export current robots.txt and CDN bot rules.
Run Promptwatch crawlability and open Agent Analytics errors on the money URLs.
Fix blocks. Re-fetch. Only then edit schema and copy.
Skip any llms.txt work that is being sold as a Google ranking task.
hello@1001seomedia.com if you want the technical pass on the retainer. Send robots.txt. Do not send a promised Overviews lift.