How to measure whether ChatGPT and other AI platforms recommend your brand

You asked ChatGPT which tools it recommends in your category. It named three competitors and left you out. Someone else on your team ran the same question an hour later and your brand came back second. Both answers were real. Neither was a measurement.

That gap is not a glitch to engineer around. It is the defining property of what you are measuring, and it should dictate your method. Teams that treat one answer as a verdict rewrite a homepage because of one unlucky sample.

Why one answer tells you nothing

A language model does not look up an answer. It samples one. Ask twice and you have drawn twice from a distribution over possible answers, not read a stored record twice. Five things move underneath you between those draws.

• Decoding is probabilistic. The model samples each token from a distribution, so the same prompt can yield different brands, in a different order, on every call.
• Retrieval shifts. Engines that search before answering — Perplexity's sonar models, or Gemini with search grounding — fetch different documents at different moments and answer from what they got.
• Models change under you. Providers update versions and default behaviour on their own schedule. A drop in your mention rate can be their release, not your content.
• Phrasing is a variable, not a constant. "Best CRM for startups" and "what CRM should a small startup use" are different prompts, and they routinely return different brands.
• Invisible context. Consumer apps personalise on memory, chat history and account state. An API call carries none of that — which makes it repeatable, and makes it a different thing from your customer's screen.

What you are actually measuring

Before the method, the disclosure — it changes how you read every number that follows. TrackGeo queries provider APIs, not consumer apps: the OpenAI API for ChatGPT, Perplexity's sonar models, the Gemini API, the Anthropic Messages API for Claude, the Azure OpenAI Service, and the Gemini API with the google_search grounding tool.

That reads like a limitation. Take it the other way. The API is the better instrument precisely because it strips out what you cannot control: no memory, no chat history, no personalisation. You query the same model that powers the assistant with the variables held still — the only condition under which a change in your number means a change in your market rather than in whose laptop ran the query.

What it will not tell you is the exact string a specific logged-in user saw on Tuesday. No external tool can. Treat API measurement as a well-controlled sample of the model's disposition toward your brand: directionally the thing your buyer meets, measured under conditions you can repeat. Per-engine surfaces and caveats are listed on /engines.

Turn answers into a rate

The unit of measurement is not the answer. It is the rate across many answers. Run a prompt n times, count the runs that mention your brand, and you have a mention rate — a number with a denominator, which is the minimum bar for anything going in a board deck.

Then measure how stable it is. For a yes-or-no signal, consistency is the share of runs agreeing with the majority outcome — mentioned in 8 of 8 is 100 percent consistent, and so is mentioned in 0 of 8, because consistently absent is still a reliable reading.

Score the confidence before you read the number

Confidence deserves to be computed rather than felt. TrackGeo blends four consistency signals, then gates the result on how much evidence backs it. The formula is deterministic — same inputs, same score, with no model asked to guess how confident it feels.

consistency = mention     x 0.40
            + competitor  x 0.25
            + citation    x 0.20
            + sentiment   x 0.15

confidence  = consistency x sampleSizeFactor(runs, engines)

The gate is the interesting half. The sample size factor runs from 0.3 to 1.0: runs contribute up to 0.7 and saturate at eight, engines contribute up to 0.3 and saturate at three. Three consequences fall out, and each is a decision you would otherwise get wrong.

• A perfectly consistent finding from one run cannot reach full confidence. The 0.3 floor grants it some credit; the sample gate withholds the rest.
• Runs saturate at eight. The ninth buys nothing — a budget instruction as much as a statistical one. Spend the next credit on a second prompt, not a ninth repeat.
• Engines saturate at three. Agreement across three engines beats agreement across thirty runs on one, because it rules out a quirk of a model rather than a quirk of a sample.

Read A is what you get from opening a chat window and asking. It looks like a 100 percent mention rate. It scores 43.1 and is labelled low. Read B is less flattering — you show up about five times in eight — and it is the one you can act on, because 76.9 confidence means the 62.5 percent is a property of the model's behaviour rather than of your sampling luck.

Confidence is not good news

This trips up nearly every team that starts measuring. High confidence does not mean you are winning. It means the reading is trustworthy. "You are absent from this prompt on every engine, 8 runs out of 8" is a maximally confident finding and terrible news. The two axes are independent, and conflating them produces the worst reporting error available: dismissing a solid, ugly number as noise while chasing a flattering one that was never real.

So read every finding as a pair: the rate tells you where you stand, the confidence tells you whether to spend money on it. Act on high-confidence bad news first — the only quadrant where you know both that there is a problem and that it is real.

Build the measurement loop

1. 1Write prompts the way buyers ask, not the way you name your category. "Best X for Y" and "alternatives to [competitor]" beat your internal product taxonomy every time.
2. 2Fix the phrasing and version it. Editing a prompt starts a new time series — do not silently splice it onto the old one.
3. 3Sample repeatedly rather than exhaustively. Eight runs on the prompts that decide deals beats one run on eighty prompts that do not.
4. 4Spread runs across at least three engines. The engine factor saturates at three, and cross-model agreement is the cheapest confidence you can buy.
5. 5Record the rate and the confidence together. A rate without a denominator and a label is a rumour with a decimal point.
6. 6Re-sample on a fixed cadence, so a change in the number means a change in the market rather than a change in when you looked.

What this cannot tell you

Measurement earns credibility by being explicit about its edges. Mention rate is not traffic. Share of answer is not pipeline. An engine recommending you is a plausible influence on a buyer, not a receipt. Correlate visibility against sessions and conversions through GA4 and Search Console — genuinely useful for prioritising work — but present it as attributed revenue and you have made a claim your data cannot support.

Once you have a trustworthy rate, two questions follow. What to compare it against — where the old SEO scoreboard quietly stops working: "AI visibility vs traditional SEO: what marketing teams need to track" (/blog/ai-visibility-vs-seo). And what to do when a competitor keeps taking the answer you want: "Why competitors appear in AI answers — and how to investigate the gap" (/blog/why-competitors-win-ai-answers). The full scoring maths is on /methodology.