Sample size: definition, formula and examples

Sample size is how many visitors each variant needs before a test can detect a given effect. The formula, a worked price test example and the common traps.

Definition

Sample size is the number of visitors each variant needs before a test can reliably detect a difference of a chosen size. It depends on the baseline conversion rate, the smallest effect you care about, the significance level and the statistical power. At 2% conversion, detecting a 20% relative change takes about 21,100 visitors per price.

  • Halving the effect you want to detect roughly quadruples the sample size.
  • The formula gives visitors per variant, so every extra price needs its own full sample.

Sample size answers "how many visitors before I can trust this?" If you test $29 against $35 on a page that converts at 2%, you need about 21,100 visitors on each price to reliably detect a 20% difference in conversion. Stop well short of that, and a real 20% difference will often be missed.

How to calculate it

For two conversion rates p₁ and p₂, at 95% confidence (two-sided) and 80% power:

Text
n per variant = (1.96 + 0.84)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²
rule of thumb: n ≈ 16 × p(1 − p) ÷ δ²

The rule of thumb comes from Evan Miller's notes on A/B testing, where δ is the absolute difference you want to detect.

Example: the baseline converts at 2%, and you want to detect a lift to 2.4%, a 20% relative change.

  • (1.96 + 0.84)² = 7.85
  • p₁(1 − p₁) + p₂(1 − p₂) = 0.0196 + 0.0234 = 0.0430
  • (p₂ − p₁)² = 0.004² = 0.000016
  • n = 7.85 × 0.0430 ÷ 0.000016 ≈ 21,100 visitors per price

Two prices need about 42,200 visitors in total. At 2,000 pricing-page visitors a month, that is 21 months.

Visitors needed per price at 95% confidence and 80% power. At 2% conversion, a 20% difference takes about 21,100 visitors per price.

Why it matters for pricing

Sample size tells you whether a price test is feasible before you start it. At the same 2% conversion, detecting a 10% difference takes about 80,700 visitors per price. Across two prices, a product with 2,000 pricing-page visitors a month would need more than six years to collect them.

So test prices further apart, where the effect you're looking for is bigger. Or use a method that limits the cost of testing, such as a bandit that moves traffic away from losing prices while the evidence builds. The price test sample size calculator runs these numbers for your traffic, and price testing with low traffic covers the options in detail.

Common mistakes

  • Computing one sample size for the whole test. The formula gives visitors per variant, so multiply by the number of prices.
  • Counting page views or sessions instead of unique visitors.
  • Stopping early because the result already looks significant, which breaks the math the sample size was based on.
  • Using conversion formulas for revenue without checking the variance. When buyers pay different amounts, through plans or add-ons, revenue per visitor varies more and needs more visitors.

Sample size is the flip side of the minimum detectable effect, and both rest on the statistical significance threshold and your baseline conversion rate. A multi-armed bandit still needs the data, but it spends less revenue gathering it.

Frequently asked questions

How many sales do I need for a price test?

Work backwards from visitors. At 2% conversion, 21,100 visitors per price is about 420 sales per price at the baseline rate, and smaller effects need many more.

Does a Bayesian test need a smaller sample?

Not for the same certainty about the same effect. Bayesian methods report results differently and handle continuous monitoring better, but separating two close prices takes a similar amount of data.