Sampling and sample size for AI visibility

Why one query proves nothing

The single most common way to be wrong about AI visibility is to ask a chatbot a question, see your competitor named instead of you, and treat that as a finding.

It is not a finding. It is one draw from a distribution. OpenAI documents chat completions as non-deterministic by default, meaning outputs may differ from request to request,[^openai-nondeterministic] and even the seed parameter, which exists precisely to make outputs reproducible, comes with the caveat that determinism may be impacted by configuration changes made on the provider's side.[^openai-seed-caveat] The system you are querying is designed to give different answers to the same question.

The consequence is worth stating arithmetically. If the true probability that a model names you for a given question is one half, a single run is a coin flip: you learn nothing about the brand and quite a lot about your own luck. If the true probability is 0.2, a single run says "no" four times out of five, which is also what a probability of 0.05 mostly says, and also what a probability of zero always says. One observation cannot distinguish between these three worlds, and they call for entirely different responses.

This is not a reason to distrust the measurement. It is a reason to do the sampling properly, which is a solved problem borrowed from every other field that estimates a proportion from a noisy process.

The unit: what exactly are you counting

Before any arithmetic, fix the unit. A run is one issue of one prompt to one model in one country, producing one answer, with a binary outcome. A cell is one combination of prompt, model and country. Depth is runs per cell per period.

Two design decisions inside that definition change what your numbers mean.

Failed runs belong outside the denominator. A timeout or a provider outage is not evidence that you were not mentioned. Excluding failures means an outage lowers your sample size rather than your rate,[^attensira-failed-runs] which is the only treatment that does not manufacture a finding out of somebody else's incident.

The daily cap. Issuing the same prompt to the same model in the same country repeatedly through a day measures the provider's caching and load balancing as much as it measures anything about you. Capping each cell at one scheduled measurement per calendar day[^attensira-daily-cap] keeps the time series interpretable at the cost of a hard ceiling on how fast depth can accumulate over time — which is exactly why depth within the day matters.

Concurrent samples measure the model; spread samples measure the world

Here is the distinction most measurement programmes never make, and it decides what your variance number is even about.

Issue three runs of the same cell at the same moment. Everything about the world is held constant — the index, the competitors' pages, the model version, the time of day. Any disagreement between the three answers is variance inside the model. That is what higher sampling depths are for: on plans above the entry tier we issue three runs per cell per day, concurrently, precisely so that the spread reflects the model rather than changes in the environment between runs.[^attensira-sampling-depth][^attensira-concurrent]

Issue three runs of the same cell spread across a week and you have measured model variance plus everything that changed in the world during that week, added together, with no way to decompose them afterwards.

Neither is wrong. They answer different questions, and a programme that never distinguishes them will misread every movement it sees. If you only ever sample once per cell per day, be honest with yourself: you do not have a variance estimate, and your week-over-week deltas contain an unknown amount of noise you cannot bound.

How wide are the error bars, really

Rates from small samples are much less certain than they look. The NIST/SEMATECH e-Handbook recommends the Wilson score interval for proportions, noting that its worth does not strongly depend on the values of n and p and that its lower limit cannot go negative — a real problem with the naive normal approximation when the count is small.[^nist-wilson]

The table below applies that published formula at 95% confidence to an observed rate of 0.20. These are computed from the handbook's formula, not measured from any dataset; they describe arithmetic, not our customers.

Read the first row again. At five runs, an observed rate of 0.20 is consistent with a true rate anywhere from about 0.04 to about 0.62. That is not a measurement; it is a rumour. And five runs per cell is more than many programmes achieve for a specific prompt on a specific model in a specific country, which is the level at which people most enjoy drawing conclusions.

The pattern in the table is the important part: the interval narrows with the square root of the sample size. Quadrupling your runs halves your error bar. This is why depth is expensive and why nobody has cheap certainty about any single cell.

How much sample does a real change need

Estimating a rate is one problem. Detecting that a rate changed is a harder one, because now the noise in two estimates has to be overcome at once.

The NIST/SEMATECH handbook's worked example is a useful anchor. To detect a shift in a proportion from 0.10 to 0.20 — a doubling — with a one-sided test at a 5% significance level and a 10% type II error rate, it requires N of at least about 102 observations, or 112 with a continuity correction.[^nist-sample-size]

Sit with that. Roughly a hundred observations to reliably detect a doubling. Required sample size scales with the inverse square of the effect size, so detecting a change half as large needs roughly four times the sample, and a change a quarter as large needs roughly sixteen times. The small, plausible, interesting movements — a rate drifting from 0.20 to 0.24 after a content project — need samples in the thousands per cell to establish. Most programmes will never have that at cell level, and should therefore make their claims at the aggregate level where the denominator is large, while resisting the temptation to narrate individual cells.

The operational form of this discipline is a gate on what gets drawn. A change is shown as a change only when it clears a two-proportion z-test at 95% confidence between the two windows; otherwise the delta is returned as not real.[^attensira-ztest] "Nothing we can prove" is then a first-class answer rather than a missing chart, and on entry-level sampling depths it will be the answer most weeks. That is the test working.

A worked hypothetical, clearly labelled

Every number in this section is invented to illustrate the arithmetic. It is not a measurement of anything.

A fictional B2B tool tracks 60 prompts on three model surfaces in one country. At one run per cell per day, a 28-day window is 60 prompts, times three surfaces, times 28 days: 5,040 runs. Suppose 1,008 of them name the brand: an aggregate rate of 0.20 on a denominator of 5,040, which by the table above carries a very tight interval. That headline number is solid.

Now the marketing lead filters to one model and one topic of twelve prompts. The denominator is twelve prompts over 28 days: 336 runs, still respectable. Then to one prompt on one model: 28 runs, where an observed 0.20 spans roughly 0.09 to 0.39. Then to that same prompt in a single week: 7 runs, where the interval covers most of the plausible range and the rate can only take eight distinct values at all.

Nothing in the product changed as they clicked. The evidence quality fell by an order of magnitude, silently, because each filter divided the denominator. This is the single most common way an honest dataset produces a dishonest conclusion, and the fix is not analytical sophistication. It is showing the denominator next to the rate at every level of filtering, so the person clicking can watch their own certainty evaporate.

Choosing depth against breadth

You have a fixed budget of runs. Spending it on more prompts buys coverage; spending it on more runs per prompt buys certainty. There is no universally right split, but there are two clear regimes.

If you are answering "how are we doing overall", favour breadth. The aggregate denominator is the sum across cells, so a wide prompt set at shallow depth gives a well-estimated headline and a defensible trend, and you simply decline to narrate individual cells.

If you are answering "did this specific change work", favour depth on the cells the change should have affected. A before-and-after on twelve prompts at depth is a real experiment. The same budget spread across two hundred prompts at depth one is a rate you cannot move and cannot interpret.

The failure mode is buying breadth and then asking depth questions of it — a wide prompt set, one run each, and a weekly meeting spent interpreting single-cell wiggles. That is the shape most AI visibility programmes actually take, and it is why so many of them produce activity without learning.

The next chapter, mention rate, citation rate and share of voice, turns to what you should be counting once you know how much of it you need.