Is llms.txt required for AI search optimization? Our view and how we test it

Clients ask us about llms.txt more than almost any other technical item. Usually someone read a post saying it is the new robots.txt and wants to know if they are behind. Our answer is calm: we have no evidence it is required, we treat it as optional, and we test it per site instead of assuming. Here is the reasoning.

For the official position of any engine, read that company's own documentation, for example Google Search Central, OpenAI's documentation on its crawlers and Anthropic's. We have not quoted any of them here, and policies change.

What it is

llms.txt is a proposed convention: a markdown file at /llms.txt that describes a site and links to the pages you most want a model to use. It is a hint. It does not grant or deny access, so it is not robots.txt. It is not a sitemap either, though it looks a bit like one.

Treating it as required gets the priority wrong. The things that decide whether you are cited are usually more basic: can the bots fetch the page, does the page answer the question, and do third parties mention you.

Our order of work

1. Access. Are the relevant AI bots allowed, and do they get successful responses? Real names to check include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User.
2. Content. Direct answers, specific details, headings that match questions.
3. Off-site presence. Mentions and citations on sources the engines trust.
4. Measurement. Prompt tracking, citations and AI-referred traffic.
5. llms.txt. Cheap to add, after the rest is in place.

Clients who publish llms.txt before fixing a blocked crawler have done step five while step one is broken.

Why we still sometimes add one

It costs little. It forces a conversation about which pages matter most. For documentation and developer-oriented sites, there are plausible consumers, since some tools can use the file as context. We write it by hand and keep it short and accurate. If it ever lists a page that no longer exists, it hurts rather than helps.

How we test it

We do not argue about it. We measure.

1. Publish the file.
2. Watch for requests to /llms.txt in crawler logs and note which bots ask.
3. Look at citation rates for the pages listed, before and after.
4. Decide whether to keep it.

We do this in Promptwatch, which we run client programs on. Agent Analytics shows real-time, IP-verified crawler logs, filtering out spoofed user agents, with stackable path filters, so we can narrow to /llms.txt and see who requests it and when. The crawl-to-citation path gives a citation rate per page, which lets us compare pages we listed against those we did not. Crawler logs start on the Professional plan ($245 a month). On Essential ($95), we read server or CDN logs instead. Promptwatch also offers free llms.txt and robots.txt generators among its free tools, which we use as a starting draft.

A single site's results prove little about another's, so we report findings as "on this site, in this period".

What the logs usually teach us

Even when llms.txt is ignored, the same logs reveal real problems: bots hitting error pages, blocked paths, pages that load only with a script, and pages that are fetched but never cited. Those findings tend to be worth more than the file ever was. The 360 Insights view joins crawler logs, citation clicks and AI-referred visits per page, which makes each page's pattern visible.

A note on training versus search bots

Some bots collect training data, some power search and some fetch pages when a user asks. Blocking the wrong ones can remove you from results you wanted. Before changing robots.txt, find out what each bot does from its operator's documentation.

Limits

We cannot prove a negative. A file that no one requests today may be requested tomorrow, and an engine could change its approach. We do not promise clients that llms.txt will lift citations, and we do not tell them it is harmless to ignore forever. We tell them it is optional, cheap and testable.

What to do this week

Check robots.txt. Read your crawler logs. If you want an llms.txt, write a short one, publish it, and look for requests after a few weeks. Then put your effort where the evidence is.