Definition
Bayesian A/B testing compares variants by updating a probability distribution for each one as data arrives, then reporting the chance that each variant is best and the expected loss of choosing it. It answers the question founders actually ask, such as how likely $39 is to earn more than $29, instead of returning a p-value.
- With a flat prior, a conversion rate's posterior is Beta(1 + buyers, 1 + visitors − buyers).
- Stop when the expected loss of choosing the leader is below the smallest difference you care about.
A Bayesian A/B test keeps a running belief about each variant and updates it with every visitor. Instead of "p = 0.41, not significant", it tells you something like: $39 has an 80% chance of earning more per visitor than $29, and if you switch now and turn out to be wrong, you'd lose about 1.2 cents per visitor on average.
How to calculate it
With a flat prior, each price's conversion rate has a Beta posterior. Draw from both posteriors many times and count:
posterior conversion ~ Beta(1 + buyers, 1 + visitors − buyers)
RPV draw = price × conversion draw
P(B is best) = share of draws where RPV_B > RPV_A
expected loss of B = average of max(RPV_A − RPV_B, 0) over all drawsExample: $29 gets 100 buyers from 4,000 visitors, and $39 gets 84 from 4,000. From 400,000 draws:
- P($39 earns more per visitor) ≈ 80%
- Expected loss of choosing $39 ≈ $0.012 per visitor, about 1.5% of its revenue per visitor
- Expected loss of choosing $29 ≈ $0.109 per visitor
These are the same numbers that give p = 0.41 in a classic test. The Bayesian readout puts them in terms you can act on: $39 is probably better, and choosing it carries a small, measurable risk.
Why it matters for pricing
Pricing decisions are about money at risk, and expected loss measures exactly that in dollars per visitor. Chris Stucchio's decision rule ends a test when the expected loss of choosing the leader falls below a "threshold of caring" (decision rule), which lets you stop early when one price is clearly ahead and keep going when it isn't.
Leverprice applies the same idea to revenue per visitor: it calls a winner when one price is at least 95% likely to be best, or when choosing the leader would cost less than 0.5% of its revenue per visitor in expectation. In the example neither condition holds yet (80% and 1.5%), so the test keeps running.
Common mistakes
- Using a strong prior without meaning to. A prior worth thousands of visitors outweighs the real data for a long time, so start from Beta(1, 1).
- Modeling conversion alone. Higher prices nearly always lose on conversion, so the comparison has to be about revenue.
- Treating a 95% probability of being best as a guarantee. It still leaves a 1 in 20 chance of being wrong, which is why the size of the expected loss matters too.
- Assuming peeking is free. In David Robinson's simulations, checking daily under an expected-loss rule made switching to a worse variant more than four times as likely; the rule keeps the average loss in check, not the error rate (Variance Explained).
Related terms
Expected loss is the stopping rule, Thompson sampling uses the same posteriors to split traffic, and statistical significance is the classic alternative. Run as a multi-armed bandit, the same posteriors also move traffic toward the leader while the test runs.
Frequently asked questions
Is Bayesian A/B testing better than the classic approach?
It answers a more useful question for business decisions and copes better with checking results often. It does not need fewer visitors to reach the same certainty about the same difference.
Which prior should I use?
A flat Beta(1, 1) for conversion rates is the usual default, and the data dominates it after a few hundred visitors. Use a stronger prior only if you have solid history for the same page and audience.