Statistical significance: definition, formula and examples

Statistical significance says how unlikely a result would be if there were no real difference. The formula, a worked price test example and the peeking trap.

Definition

A test result is statistically significant when a difference as large as the one observed would rarely appear by chance if the variants actually performed the same. Most tests use a 5% threshold, so the p-value must fall below 0.05. Significance says nothing about how large a difference is or how likely a variant is to be better.

  • A p-value below 0.05 means the data would be unusual if there were no real difference.
  • Checking a test repeatedly and stopping at the first significant result inflates false positives.

Statistical significance is a filter against luck. Suppose $39 earns $0.82 per visitor and $29 earns $0.73 after 4,000 visitors each. Significance asks: if the two prices really performed the same, how often would random variation alone produce a gap this big? If the answer is rarely, under 5% of the time by the usual convention, the result is called significant.

How to calculate it

For revenue per visitor, compare the two means with a z-test (a t-test gives the same answer at these sample sizes). When every buyer pays the list price, the variance per visitor follows from the price and the conversion rate:

Text
z = (RPV_B − RPV_A) ÷ √(s_A² ÷ n_A + s_B² ÷ n_B)
s² = price² × p × (1 − p), where p = conversion rate
two-sided p-value = 2 × (1 − Φ(|z|))

Example: $29 gets 100 buyers from 4,000 visitors (2.5%), and $39 gets 84 (2.1%).

  • RPV: $0.725 against $0.819, a 13% gap
  • Variances: 29² × 0.025 × 0.975 = 20.5 and 39² × 0.021 × 0.979 = 31.3
  • z = 0.094 ÷ √(20.5 ÷ 4,000 + 31.3 ÷ 4,000) = 0.83
  • p-value ≈ 0.41

A gap this size would appear about 41% of the time with no real difference, so it isn't significant. The test needs many more visitors before a 13% difference can clear the bar.

Why it matters for pricing

Price decisions are hard to undo, because customers who signed up at a price expect to keep it. Significance protects you from locking in a price on the strength of a lucky week.

The 5% only holds if you fix the sample size in advance and look once. In Evan Miller's example, running a significance test after every observation and stopping at the first significant result produced false positives 26.1% of the time instead of 5% (How Not To Run an A/B Test). If you check results daily, a method built for continuous monitoring fits better, such as a Bayesian test with an expected loss stopping rule or a multi-armed bandit. Leverprice works that way: it calls a winner when one price is at least 95% likely to be best, or when the remaining prices are practically equivalent.

Common mistakes

  • Peeking at the results and stopping as soon as they look significant.
  • Reading p = 0.04 as a 96% chance that the new price is better. The p-value assumes there is no difference and measures how surprising the data is; it isn't the probability that either price wins.
  • Testing significance on conversion rate when the decision is about revenue. In the example, conversion favors $29 while revenue per visitor favors $39.
  • Ignoring the size of the effect. With enough traffic, a 0.5% difference becomes significant without being worth acting on.

Sample size and the minimum detectable effect are fixed before a significance test starts, and the comparison needs a control group running at the same time. Bayesian A/B testing replaces the p-value with the probability that each variant is best.

Frequently asked questions

What does 95% significance mean?

It means the test used a 5% threshold: if there were no real difference, a result this extreme would appear less than 5% of the time. It does not mean there's a 95% chance the variant is better.

Can I stop a test as soon as it becomes significant?

Not with a classic fixed-sample test. Stopping at the first significant reading inflates false positives, so either commit to a sample size up front or use a method designed for continuous monitoring, such as a sequential or Bayesian test.