How an AI visibility score is calculated, and how to tell if a tool's metrics are trustworthy
What goes into an AI visibility score, where the numbers mislead, and the checks we run on a tool's metrics before we report them to a client.
Clients ask us to explain their AI visibility score more than any other number we report. They want to know what it is made of, and whether they can trust it in a board deck. Our honest answer starts with this: there is no industry formula. Every tool builds its own, so the first job is to understand the one you are looking at.
What the number is made of
Under the hood, a visibility score is a ratio from a sample. The sample is a set of prompts run against a set of engines. The ratio counts something: how often your brand is named, how high it appears, how much of the total conversation it takes up.
In our work we break it into parts and report them separately, because one blended number hides what moved.
- Visibility Score: the headline measure of how often you appear for the tracked prompts.
- Share of Voice: your mentions against competitors' mentions across the same prompts.
- Position in Answer: first recommendation or last item in a list.
- Sentiment: how the engine describes you, scored per response from 1 to 100 in Promptwatch, with a weekly summary.
We run client programs on Promptwatch, which keeps these as distinct metrics and runs prompt panels daily per model, country and language. That structure is what lets us explain a movement instead of just reporting it.
Where we have seen scores mislead
We would sooner flag these early than have a client find them later.
A convenient prompt set. If the prompts are easy, the score is flattering. We write prompt sets with the client, split by topic, persona and country, and keep branded questions apart from non-branded ones. Branded prompts will almost always score well and say little about discovery.
A single run. Answers change between runs. We never present one check as a finding. We look at a daily panel over several weeks.
Interface confusion. A raw API response is not the same thing as what a person sees in ChatGPT Search. Promptwatch monitors the live interfaces for ChatGPT (Search), Google AI Overviews, Google AI Mode, Copilot, Gemini, Perplexity, Alexa and Meta AI, and covers Claude, DeepSeek, Mistral and Grok through APIs. We tell clients which is which and avoid wording that suggests every engine is read from a consumer screen.
A loose definition of "mentioned". Brand aliases and a Brand Book, whose instructions decide what counts as a mention, keep us honest. Without them a brand with a common-word name can be wildly over or under counted.
The checks we run before trusting a tool
When we evaluate any platform, we ask a short list of questions and write down the answers.
- Which engines are real-interface and which are API?
- How often do prompts run?
- Can we see the individual responses behind the number?
- Can we control the prompt set and the mention definition?
- Can we export the data? Promptwatch offers Excel and CSV exports with a column picker, so we can check totals ourselves.
- Does a manual spot check agree? We take five prompts, run them by hand and compare.
The sixth one catches the most. If a tool's count disagrees with what you see yourself, ask why before you report it.
Check the score against data you own
This is the step most teams skip, and it is the one we rely on. A visibility score describes what the engines said. Your server logs and analytics describe what happened. Promptwatch lets us put them side by side. Agent Analytics gives real-time, IP-verified AI crawler logs, with a crawl-to-citation path and a citation rate per page. Visitor analytics shows AI-referred sessions and conversions, installed through a lightweight script, a GTM template or the WordPress plugin.
When the score and the traffic agree, we are confident. When they diverge, we dig in. A rising score with flat AI-referred visits can mean the prompts are not the ones that drive real buyers. A falling score with stable visits can mean sampling noise.
A tracker that gives you only a score gives you no way to run this check. That is the main reason we prefer a platform that also has crawler logs and visitor analytics.
What we tell clients
We say four things plainly. The score is a proxy, not revenue. It is built from a sample, and we show the sample. It will move for reasons outside the client's control. And no tool can promise a particular score, since nobody controls what the engines say.
We also keep the prompt set stable. If a client wants to change it, we note the date on the report so the trend line carries an explanation.
A practical way to start
If you are choosing a tool now, run the same ten prompts in two of them and by hand. Compare the three. Differences will show you where each tool's method departs from reality, which is more useful than any vendor explanation. Then pick the platform whose numbers you can trace.