Expected loss: definition, formula and examples

Expected loss is the average amount you give up by choosing a variant that turns out not to be best. The formula, a worked example and how to set a threshold.

Definition

Expected loss is the average amount you would give up by choosing a variant, combining how likely it is to be worse than the best option with how much worse it would be. Bayesian A/B tests use it as a stopping rule: once the leader's expected loss falls below the smallest difference you care about, the test can end.

  • Expected loss of a variant = average over posterior draws of (best draw − that variant's draw).
  • A 60% chance of being best can still be a safe choice when the possible shortfall is tiny.

Expected loss is the cost of being wrong, averaged over how likely you are to be wrong. Choosing $39 over $29 might be a mistake with 20% probability. If, when it is a mistake, it costs half a cent per visitor, the expected loss is about a tenth of a cent per visitor, which is small enough to stop testing and charge $39.

How to calculate it

Draw many plausible values for each variant from its posterior distribution, then average the shortfall:

Text
expected loss of A          = average over draws of (max of all draws − draw of A)
relative expected loss of A = expected loss of A ÷ expected value of A

Example with five posterior draws of revenue per visitor (real engines use thousands):

Draw$29$39BestLoss if you pick $39
1$0.70$0.82$0.82$0.00
2$0.76$0.74$0.76$0.02
3$0.72$0.85$0.85$0.00
4$0.78$0.77$0.78$0.01
5$0.69$0.80$0.80$0.00

$39 wins 3 of the 5 draws, so it is 60% likely to be best. Its expected loss is ($0.02 + $0.01) ÷ 5 = $0.006 per visitor. Its average draw is $0.796, so the relative expected loss is 0.75%. Picking $29 instead would carry an expected loss of $0.072 per visitor.

Why it matters for pricing

Expected loss turns "are we sure?" into "how much could this cost?", and the second question is the one you can act on. Multiply it by the traffic you expect: $0.006 per visitor across 50,000 visitors a month is $300 a month of expected shortfall, which tells you whether another week of testing is worth it.

It also ends tests that would otherwise run forever. When two prices earn almost the same, neither ever becomes clearly best, but the expected loss of picking either one keeps shrinking, and stopping becomes safe. Chris Stucchio called this cutoff a "threshold of caring" (decision rule). Leverprice uses a relative cutoff of 0.5%: once choosing the leader would cost less than 0.5% of its revenue per visitor in expectation, the remaining prices count as practically equivalent and the leader is called the winner.

Common mistakes

  • Confusing expected loss with the probability of being wrong. A 40% chance of a tiny shortfall can be safer than a 5% chance of a large one.
  • Using one absolute threshold for every product. $0.01 per visitor is noise at 1,000 visitors a month and real money at a million.
  • Setting the threshold after looking at the results.
  • Computing it on conversion rate when the decision is about revenue.

Expected loss is the stopping rule of Bayesian A/B testing and comes from the same posteriors as Thompson sampling. It does the job that statistical significance does in a classic test, and for prices it is measured in revenue per visitor.

Frequently asked questions

What threshold of expected loss should I use?

The smallest shortfall you'd shrug off, either in absolute terms (cents per visitor times your monthly traffic) or relative to revenue. A relative threshold such as 0.5% scales with your price automatically.

Is expected loss the same as regret?

They're related. Regret is the revenue already lost by showing worse variants during a test, the cost a multi-armed bandit tries to keep small. Expected loss looks forward and estimates what you would give up by choosing a variant now.