How do I use an llms.txt file, and does it help AI search visibility?
We publish llms.txt files for clients but never promise a lift. Here is how to write one, what we tell clients, and how we test whether bots read it.
Almost every client asks about llms.txt in the first month. The question usually arrives with a vendor claim attached, promising it will lift AI visibility. Our position is simple: write one, because it is cheap, and do not expect anything until you have evidence from your own logs.
What it is
llms.txt is a proposed convention. You put a markdown file at the root of your site that gives language models a short, curated guide: the brand name, a one- or two-sentence summary, then sections of links with a line on each page. Some sites add a longer llms-full.txt with fuller reference content.
It is a proposal, not a long-standing standard like robots.txt. We cannot point to a verified statement from the major AI or search vendors that their assistants read it and use it to choose citations. So when someone tells you it boosts visibility, ask what they measured.
Why we still write it
The cost is an hour or two. The risk is nil if the contents are accurate. And the exercise has side benefits. To write the file you have to decide which pages are your authoritative ones, and that conversation often exposes outdated pricing pages, duplicate guides and orphaned documentation. That cleanup helps regardless of whether any bot reads the file.
How we write it
- Heading and summary. The brand name, then a plain statement of what the company does and who it serves.
- Sections. Products, pricing, documentation, guides, company. We link only pages we would want treated as the source of truth.
- One line per link. What the page contains, without sales language.
- Facts matched to the site. Plan names, pricing and product names must be identical to the live pages. A contradictory file is worse than none.
- Published at the root as plain text with a normal 200 response.
- Reviewed whenever pricing or positioning changes.
We use the free llms.txt and robots.txt generators from Promptwatch as a starting draft, then edit by hand. Promptwatch also publishes its own llms.txt and llms-full.txt, which is a reasonable reference for format.
Do the unglamorous part first: robots.txt and access
Before llms.txt, we check whether the AI crawlers can reach the client's pages at all. robots.txt is the file crawlers are built to honor, and different bots do different jobs: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and others. Whether to allow each is a business decision, and we put it to the client explicitly. If the aim is to be recommended in search-style answers, blocking the search-related bots by default works against it.
The more common problem is accidental blocking. A CDN or firewall rule can quietly refuse bots that robots.txt allows. We have learned to look at the logs rather than trust the config.
How we test whether llms.txt does anything
We do this properly or not at all.
- Publish the file and note the date.
- Watch the crawler logs for requests to /llms.txt, by bot.
- Track citations for the relevant prompts before and after.
- Change nothing else at the same time, so any movement can be attributed.
The tool for this is Agent Analytics in Promptwatch. It shows real-time, IP-verified AI crawler logs, so spoofed user agents are filtered out. Stackable path filters let us isolate /llms.txt, error tracking explains failures, and a crawl-to-citation path shows a citation rate per page. Citation Trends over time tells us whether anything moved. Logs can arrive through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP source or the WordPress plugin. Crawler logs start on the Professional plan, and we make sure clients know that before they buy a tier below it.
If the logs show that no bot ever requests the file, we tell the client plainly, and the file stays as a harmless reference rather than a strategy.
What we tell clients to prioritize instead
- Crawl access. Pages fetched without errors by the bots that matter.
- Answers up front. Pages that state the answer to a buyer question plainly near the top.
- Prompt coverage. Content for the prompts the client actually cares about. The content gap analysis shows which prompts lack a good page.
- Third-party presence. Reviews, mentions and citations on other sites.
None of these needs a new file format, and all of them are measurable.
The expectation we set
An accurate llms.txt is good housekeeping. It might help, it might do nothing, and the logs will say which. No file can guarantee citations, and no vendor promise should be treated as evidence. We ask clients to judge it the way they would judge any other change: by the data on their own site.