llms.txt: What We Tell Clients to Do About It

Ask ten people in this industry what llms.txt does and you will hear ten answers, some of them confident. A few years ago it did not exist. Now clients ask about it in the first call, usually because someone on LinkedIn told them it was the missing piece.

Our advice is plain: it is cheap, it is harmless, and it is not the thing that will move your numbers. Here is the longer version.

What the file is

llms.txt is a plain text file, usually in Markdown, placed at the root of a site. It is meant to give language models a readable summary of the site and links to the most useful pages. Think of it as a curated table of contents written for machines, as opposed to a sitemap that lists everything.

You can see a working example at promptwatch.com/llms.txt, which includes an overview, plans, features, and links, and is dated so readers know how current it is. There is also an expanded version for detail.

What we do not claim

We do not tell clients that adding the file will make them appear in AI answers. We have no evidence that it does, and we do not accept vendor claims on the subject without testing. No major engine has, to our knowledge, said that it uses the file as a ranking signal. If that changes, the vendor's documentation is where to look.

We also do not tell clients that the file is useless. A well-written summary helps any agent or person who reads it, and some tools and agents do fetch it. The honest position sits between hype and dismissal.

Why we still recommend writing one

It forces clarity. To write the file you have to decide what the five or ten most important pages are and how you describe the company in a paragraph. Many clients have never done that. The exercise is valuable even if no bot reads the result.

It gives agents a clean source. When an agent visits a site to answer a question, a short, accurate summary is easier to use than a navigation menu and a cookie banner.

It is a single place to keep facts current. Prices, product names, and dates change. A dated summary file makes stale claims easier to catch.

It costs almost nothing. A few hours at most, and it can be generated from existing content.

How we write one

1. Start with a plain description. Two or three sentences about what the company is and who it serves. No slogans.
2. List core pages with one-line descriptions. Product, pricing, documentation, key guides, and policies. Describe each in neutral terms.
3. State facts we can verify. Names, locations, and founding dates where they exist. We do not include numbers we cannot source.
4. Add a date. "Last updated" lets readers judge freshness.
5. Keep it short. The point is curation. If the file is longer than a couple of screens, it has become a sitemap.

Free generators exist, including the llms.txt and robots.txt generators from Promptwatch's free tools. We use them as a first draft and then edit by hand.

What matters more

If a client has limited time, we push these ahead of llms.txt:

• Crawler access. If the bots cannot fetch the pages, nothing else matters. Check robots.txt, CDN rules, and response codes.
• Clear page-level answers. A page that states the answer in the first paragraph is easier for any engine to use.
• Accurate, consistent facts across the site and across third-party listings.
• Pages that match the prompts buyers use.

Those are measurable. You can see in crawler logs whether bots fetch the pages and in citation data whether they cite them.

How we test whether the file matters

We do not take it on faith, so we check. After publishing the file we look at the logs for requests to /llms.txt. In Promptwatch's Agent Analytics, path filters make it easy to isolate a single URL and see which bots requested it, and when. If nobody asks for the file, we learn something. If a particular bot does, we learn something else.

Then we compare citation data across a period. If nothing changes, we say so. If something does, we remain cautious about cause, because we usually changed other things too. We would rather say "we saw requests and no measurable effect yet" than invent a result.

Markdown versions of pages

A related question is whether to serve Markdown copies of pages for agents. We treat that as a separate technical decision, with its own maintenance cost. It is worth considering for documentation-heavy sites. For most marketing sites, improving the HTML is the better use of time. Whatever the choice, keep the content identical to the HTML version, because divergence invites mistakes.

Common mistakes

• Treating the file as a keyword stuffing opportunity. It will read badly and help nobody.
• Listing every URL.
• Letting it go stale after a pricing or product change.
• Including claims that appear nowhere else on the site.
• Announcing it to the client as a growth lever.

What we put in the client note

Two lines. "We added a short, dated llms.txt. We are monitoring requests to it and will report any effect honestly." That sets expectations at the right level.

Our recommendation

Write the file, keep it short and dated, and then go and spend your real effort on crawler access and page-level answers. Check the logs to see whether anyone reads it. We do that check with Agent Analytics in Promptwatch, which shows crawler requests by path. If you want to generate a first draft, start at promptwatch.com.