llms.txt and Markdown: What We Tell Clients
A practical client brief on llms.txt and Markdown for AI search, grounded in the markdown-in-ai-search report. What these formats do, what they do not do, and where the real work is.
Twice a month a client sends us a link about llms.txt. The link is usually a vendor explaining that a single Markdown file at the root of their site will make them visible in ChatGPT. The client asks whether they should do it. The honest answer is: yes, spend an afternoon on it, and no, do not expect it to move your visibility. This is the brief we give them, written for the marketing director who has to decide what to do with the next LinkedIn post about Markdown.
We run client programs on Promptwatch. The reason that matters here is that we measure what these formats actually do. We do not have to argue from theory. We can show a client the citation trend before and after they shipped an llms.txt, and the trend is usually flat. That is not because llms.txt is useless. It is because the thing that moves visibility is almost never the file format. It is the crawl, the content, and the citations, and those are the parts nobody puts in a viral post.
What llms.txt actually is
llms.txt is a convention for a Markdown file served at the root of a domain, intended to give language models a concise, structured summary of what the site is and where its important content lives. Think of it as a sitemap written for a reader rather than a crawler. It lists the product, the audience, the key pages, and links to the documentation or the canonical content, all in plain Markdown.
The format is reasonable. A model that fetches the file gets a clean, token-efficient summary instead of having to reconstruct the site from HTML. For documentation-heavy sites, software vendors, and platforms with a lot of structured reference material, an llms.txt is a genuine courtesy to the models that read it. We write them for clients who have that kind of content. We do not write them as a visibility tactic.
The reason we do not write them as a visibility tactic is the data. The markdown-in-ai-search report tracks how often Markdown appears as a citation source in AI answers. The figure is 0.05% of citations. That is not a typo. Markdown is a rounding error in the citation mix. The formats that actually carry citations are normal HTML pages, Reddit threads, YouTube videos, and offsite mentions on publisher domains. A client who ships an llms.txt and waits for citations to climb will wait a long time.
What Markdown is good for in AI search
Markdown still has a place. It is just not the place the vendor posts imply.
First, Markdown is the cleanest format to feed a model directly. When we build a Content Agent pipeline, the drafts the agent produces are Markdown, and the review inbox in Promptwatch shows them as Markdown. That keeps the agent's output legible and editable before it goes to the CMS. Markdown is a working format inside our process, not a deliverable we hand to a search engine.
Second, Markdown is a useful target for content that is meant to be quoted. A clean Markdown doc with clear headings and short factual paragraphs is easier for a model to lift a sentence from than a heavy single-page app. We recommend Markdown for changelogs, API docs, pricing appendices, and the kind of reference pages a buyer might ask an engine to summarize. That is a content decision, not a file-format decision.
Third, llms.txt is a useful index for the models that fetch it. If a client has a real documentation surface, an llms.txt that points at the canonical pages helps a model find them faster. The effect is on the crawl side, not the citation side. We measure it with the crawler logs in Promptwatch Agent Analytics, which record the AI crawlers that hit the site, ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, and the Meta AI crawler. When an llms.txt is doing its job, we see the relevant crawlers hit the linked pages. When it is not, we see nothing change.
What we tell clients to do with llms.txt
We give clients a short, dated checklist. It is deliberately small.
Ship an llms.txt if you have documentation, a product with a real reference surface, or a site where the canonical content is hard to extract from the chrome. Keep it under a few hundred lines. Link the pages you would hand a new hire on their first day. Use plain Markdown, no clever extensions. Date the file and update it when the product changes, not on a weekly content treadmill.
Do not ship an llms.txt as a replacement for fixing the site. If the site is uncrawlable, the llms.txt is uncrawlable too. If the canonical pages return 5xx or block the AI crawlers in robots.txt, the llms.txt will not rescue them. We have seen clients block ChatGPTBot and PerplexityBot in robots.txt and then ship an llms.txt the next week. The file is irrelevant if the crawlers are not allowed in.
Do not pay a vendor for an llms.txt strategy. It is an afternoon of work for a writer who knows the product. The strategy is the content the file points at, not the file.
Where the real work is
The real work is the part that does not fit in a LinkedIn post. We freeze a prompt list for the client, pulled from their search console data and the query fanouts report, so the visibility trend is readable. We pull the crawler logs and fix the pages that are cited but misfiring. We run content gap analysis on the prompts where the client is absent and a competitor is cited, and we draft to fill those gaps, with a human reviewing every slot before publish. We track visitor analytics so the client can argue from pipeline, not from a visibility score.
None of that is as easy to sell as "ship a Markdown file." All of it is what actually moves the numbers. The Markdown file is a courtesy. The program is the work.
What we will not promise
We will not promise that an llms.txt will increase citations. The data does not support it. We will not promise a visibility lift from Markdown alone. We will not sell a client a file-format strategy when they need a crawl and content strategy. The commitments we keep are the boring ones: no long-term contracts, senior specialists only, and a review inbox a human actually opens.
If you want the brief above turned into a program on your site, with the file shipped correctly and the crawl and content work behind it, write to us at hello@1001seomedia.com. We will scope the prompt list, pull the crawler logs, and show you what moves and what is just a file at the root.