How many AI answers make a sample?

How thin are real samples in a working programme?

Thinner than most dashboards let on. Here are real counts from our own workspace, which tracks attensira.com on ChatGPT and Perplexity and takes three samples per daily reading, issued together (n=3). They are the 30 days to 4 October 2026, read over the Attensira MCP that morning. The intervals in the table at the top of this page are 95% Wilson intervals, computed with the formula in the NIST/SEMATECH handbook.

Read the first row slowly. Ahrefs was named once in six answers. That is consistent with Ahrefs being named almost never or in more than half of all answers. The question was added to our set in late September, so it has six answers instead of the fifty-odd our older questions have. Nothing about it is wrong; there is just not enough of it yet.

Then read the last two rows. Pool every question and the interval tightens to under a point either side. Those are numbers you can hold a meeting about, provided you also know what they are an average of, which the previous chapter covered.

The pattern is the whole chapter in miniature: single questions on single engines are noisy, pooled numbers are steady, and the count beside the rate tells you which one you are looking at.

Why does one answer prove nothing?

Because the system is built to vary. OpenAI documents its chat completions as non-deterministic by default: the same request can produce different outputs. Consumer surfaces add search results that change by the hour. Ask the same question twice and you can get two different shortlists.

The arithmetic is unforgiving. If the true chance that an engine names you is one in two, a single answer is a coin flip. If it is one in five, a single answer says "no" four times out of five, which is also what one in twenty mostly says and what zero always says. One observation cannot tell those worlds apart, and they call for completely different responses.

This is not a reason to distrust measurement. Estimating a proportion from a noisy process is a solved problem in every other field. It is a reason to sample properly and to show the sample.

What exactly is being counted?

Fix the unit before the arithmetic.

• A run is one question sent to one engine for one country at one moment, producing one answer. It either named you or it did not.
• A cell is one combination of question, engine and country.
• Depth is runs per cell per day.

Attensira runs each cell at most once a day, on that question's own daily slot. At each slot it takes one sample on Starter and three on Growth and Business. An answer that names you nine times still counts once.

Two design decisions inside that definition change what your numbers mean.

Failed runs are left out. A timeout or a blocked request is not evidence you were not named. Attensira stores failed runs for debugging and excludes them from the count. In our month, 61 ChatGPT runs and 44 Perplexity runs failed; the sample shrank and no rate moved.

A daily cap keeps the series readable. Asking the same question again and again through a day measures the engine's caching as much as anything about you. One slot a day, with the depth taken at that slot, keeps each day comparable with the next.

Should samples be taken together or spread out?

They answer different questions, so choose on purpose.

Issue three runs of the same cell at the same moment and everything about the world is held still: the index, the competitors' pages, the model version. Any disagreement among the three answers is the engine's own variance. That is why Attensira issues the three samples per daily reading on Growth and Business together (n=3).

Issue three runs spread across a week and you have measured the engine's variance plus everything that changed in the world that week, added together, with no way to separate them afterwards.

A programme that never distinguishes the two will misread every movement. If you only ever sample once per cell per day, be honest about it: you have no estimate of the engine's own variance, and your week-on-week changes contain noise you cannot bound.

How wide are the error bars at normal sample sizes?

Wider than the decimals suggest. The NIST/SEMATECH handbook recommends the Wilson score interval for proportions, partly because its lower limit cannot go negative, which matters when counts are small. The frame below applies it to a fixed rate of 0.20 at growing sample sizes. The interval narrows with the square root of the sample. That is why depth is expensive and why nobody has cheap certainty about one question on one engine.

Our own data sits right on that table. Our questions tracked since mid-September had between 42 and 63 answers per engine in the month, which is the 50-answer row: good enough to tell "rarely" from "usually", not good enough to tell 0.20 from 0.28.

This is also why Attensira shows the count with every rate, switches the Overview headline to the count below 20 answers, and marks counts under 20 as thin in the prompts grid. "1 of 6" carries its own warning. The same number written as a percentage does not.

How many answers does a real change need?

More than estimating a rate, because two noisy estimates have to be told apart.

The NIST/SEMATECH handbook's worked example is a good anchor. To detect a proportion moving from 0.10 to 0.20, a doubling, with a one-sided test at a 5% significance level and a 10% chance of missing a real change, it needs at least about 102 observations, or 112 with a continuity correction.

Roughly a hundred observations to catch a doubling. Required sample size grows with the inverse square of the effect, so a change half as big needs about four times the sample. The small, plausible movements a content project produces need hundreds of answers per cell to establish, which is why per-question claims should be rare and pooled claims common.

Attensira turns this into a gate. A change is drawn as a change only when it clears a two-proportion z-test at 95% confidence against the previous window of the same length, and a window with fewer than two runs on either side is never a change. Otherwise the delta comes back as "nothing we can prove yet". On one sample a day, that will be the answer most weeks. That is the test working.

Our own numbers show the gate in both directions. Attensira's mention rate on ChatGPT, 90 of 2,274, and on Perplexity, 90 of 2,100, look different to two decimals; the z-score between them is about -0.6, nowhere near the bar. The two engines name us equally often as far as this sample can tell. Our citation rates, 142 of 2,274 on ChatGPT and 77 of 2,100 on Perplexity, give a z-score near 3.9: that difference is real.

Can a significant change still be wrong?

Yes, and our own chart has the example.

On 23 September 2026 our daily ChatGPT mention rate was 0 of 81 answers. On 24 September it was 9 of 132. Run the two-proportion test on those days and it clears the bar with a z-score of about 2.4. Statistically, something moved.

What moved was our prompt set. That was the day our three branded questions, "what is Attensira", "Attensira reviews" and "Attensira vs Profound", began producing answers: three questions, three samples each, nine answers that named us. Nothing about how engines treat attensira.com on unbranded questions changed that day, or since.

A significance test answers one question: is this difference bigger than the noise? It cannot tell you whether the two windows measured the same thing. That is a job for a dated change log, and for comparing only the questions present on both sides, which the prompt-set chapter covers.

Should you buy depth or breadth?

You have a fixed number of answers a month. Spending them on more questions buys coverage; spending them on more runs per question buys certainty. Two regimes are clear.

If the question is "how are we doing overall", favour breadth. The pooled count is the sum across cells, so a wide set at shallow depth gives a steady headline. Then decline to narrate individual cells.

If the question is "did this change work", favour depth on the questions the change should affect. A before-and-after on twelve questions with three samples per daily reading, issued together (n=3), is a real experiment. The same budget spread across two hundred questions read once is a rate you cannot move and cannot interpret.

The failure mode is buying breadth and asking depth questions of it: a wide set, one run each, and a weekly meeting spent explaining single-cell wiggles. Show the count next to every rate at every level of filtering, and the people in that meeting can watch their certainty disappear as they click.

The next chapter, mentions, citations and share of voice, turns to what you should be counting once you know how much of it you need.