Measuring LLM visibility: mention rate, not rank

How do you measure LLM visibility?

Ask a fixed set of realistic buyer questions to each AI provider, repeatedly, and record the fraction of answers that mention your business. Report that mention rate with a confidence interval, per provider, keeping grounded (web-search) and model-memory answers as separate metrics.

Every element of that sentence is load-bearing: fixed questions make runs comparable, repetition tames randomness, intervals keep you from chasing noise, and the grounded/memory split tells you which fix to apply when the number is bad. The same rigor shows up in the academic side of the field — the original GEO benchmark (KDD 2024) evaluates visibility as an impression share across many queries, never a single-answer rank.

What is a mention rate?

The share of probed answers that name your business: mentioned in 12 of 40 answers is a 30% mention rate. It is the LLM-era analogue of a ranking report — except it honestly reflects that answers are generated fresh each time rather than read off a stable index.

• Count a mention when the assistant names you — optionally track position-in-answer and cited URLs separately.
• Track competitors in the same runs; share-of-voice context makes the number actionable.
• Never blend providers into one score: a ChatGPT gain can hide a Gemini collapse.

Why does the question set need to be frozen?

Because if the questions change between measurements, you measured the questions, not your progress. A frozen set — same wording, same order of magnitude, versioned like code — is the only way a month-over-month comparison means anything.

GEOExtension freezes and hashes its question sets so any two reports can prove they used identical inputs. When you do add or reword questions (markets change), version the set and start a new baseline rather than silently mixing eras.

Why report confidence intervals?

Because LLM output is stochastic, a mention rate from a finite sample is an estimate, not a fact. A 30% rate from 40 questions carries a 95% Wilson score interval of roughly 18–45% — so a “jump” from 30% to 35% next month is statistically indistinguishable noise.

The Wilson interval is preferred over the naive ±z√(p(1−p)/n) formula because it behaves sensibly at small samples and extreme rates (0% or 100%) — exactly the regimes small-business probes live in; Evan Miller’s classic essay on rating math shows the same failure mode in another domain. Interval width shrinks with roughly the square root of sample size: to halve your uncertainty, quadruple your questions.

Why measure grounded and model-memory answers separately?

Because they are different systems with different failure modes and different fixes. Grounded answers reflect what retrieval bots can fetch today; model-memory answers reflect what training runs absorbed months ago. One blended score would tell you something is wrong without telling you what to do.

• Low grounded, decent memory — an access problem: check robots.txt, JavaScript-only content, and your presence on cited platforms.
• Decent grounded, low memory — an entity problem: your public footprint was thin or inconsistent when models trained. Fix the durable sources and wait for releases.
• Both low — start with access (it is faster to fix), then build the citation footprint the answer-assembly process actually reads.

What should you refuse to trust?

Any metric that would be impossible given how LLMs work: a stable “AI rank #3”, a visibility score with no sample size, one-screenshot proof of “being recommended”, or a single blended number across providers and answer modes. If a report cannot say how many probes produced it, it is marketing, not measurement.