What is LLM tracking and how does it work? Our practitioner view
LLM tracking logs how AI engines name your brand across repeated prompts. Here is how we set it up for clients, what it misses, and where it goes next.
When a client asks us what LLM tracking is, we skip the definition and show them a spreadsheet. Column one is a buyer question. The other columns are engines. Each cell says whether the brand was named that day, and which pages were cited if so. LLM tracking is that spreadsheet, automated and run every day.
A note on terms. Some engineers use "LLM tracking" for observability of model calls inside an app. We mean the marketing version: tracking how brands appear in ChatGPT, Gemini, Perplexity and similar engines.
What we actually measure
We keep the metric set small, because clients lose interest in dashboards with forty tiles.
- Visibility: the share of tracked answers in which the brand is named.
- Share of voice: the brand against named competitors across the same prompts.
- Position in the answer: first mention versus the fourth.
- Citations: which URLs the engine used as sources, ours or someone else's.
- Sentiment: how the answer describes the brand.
Inside Promptwatch these map to Visibility Score, Share of Voice, Position in Answer, citation analytics and a per-response sentiment score. It is the platform we run client programs on, partly because it keeps all five in one workspace.
How we build a prompt set
This is where most programs go wrong. A prompt list copied from keywords produces numbers that look precise and mean little.
We start from the client's sales calls, support tickets and search data, and write prompts in the way a buyer would phrase them to a chatbot. Then we tag them: brand, category, competitor comparison. We add personas, because "best CRM for a solo consultant" and "best CRM for a sales team of 40" return different brands. Promptwatch supports persona stacking, so one prompt can be run as several buyer types. Prompt Explorer can suggest prompts from Search Console data, ranking keywords or People Also Ask, which saves the blank-page problem.
We also set the Brand Book so the tool knows what counts as a mention. A brand with a common-word name needs aliases and rules, or the counts are wrong.
Why daily runs matter
AI answers vary between runs. One check is a coin flip. We want daily samples so we can look at a trend across two or three weeks and ignore single-day swings. When a client panics about a drop on Tuesday, the trend line is usually what calms the conversation, or it confirms the drop is real.
API data versus the real interface
Some trackers call model APIs. Others read what the consumer product shows. They are not the same thing, because the consumer product adds live search and its own instructions. Promptwatch monitors ChatGPT Search, Google AI Overviews, Google AI Mode, Copilot, Gemini, Perplexity, Amazon Alexa and Meta AI through their real interfaces, and tracks API models such as Claude, DeepSeek, Mistral and Grok alongside. We tell clients which engines fall into which group, because it changes how much weight a number deserves.
What mention tracking cannot tell you
A tracker answers "were we named?" It does not answer "why not?" For that we move to three other data sources.
- Citation analytics. Which pages and domains win the answer on the prompts we lose. Offsite mentions matter here: a brand named on third-party pages without a link still shapes answers.
- Crawler logs. Agent Analytics shows IP-verified hits from bots like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. If an engine's crawler gets errors on your key pages, content changes will not help until that is fixed. Crawler logs start on the Professional plan.
- Visitor analytics. Whether any of the visibility becomes visits and conversions.
We often find the answer to a visibility problem sits in the second source, not the first. A monitoring-only tool never shows it.
Limits worth stating to clients
No tool can promise a recommendation. We say so at kickoff. Tracking tells you where you stand and what changed; it does not buy you a slot in an answer. We also avoid treating an absolute visibility number as a target. The direction over time, against competitors, is the useful part.
Turning the data into work
Each week we pull the prompts where a competitor is named and the client is not, read the citations on those answers, and decide among three actions: fix a page, create a page, or earn a third-party mention. The Action Board in Promptwatch generates a prioritized list from the data, and the Discovery agent proposes evidence-backed actions on a weekly schedule. We review them rather than accept them blindly.
Where to start
If you are starting from zero, begin with 25 to 50 prompts, three or more engines, and two weeks of daily data. Do not change content in the first week, or you will not know what caused what. After two weeks, read the citations, pick one gap and fix it. Then watch the trend. That small loop is LLM tracking in practice, and it is what we repeat for every new client.