Thompson sampling: definition, formula and examples

Thompson sampling picks which variant to show by drawing from each one's probability distribution. The formula, a worked pricing example and common mistakes.

Definition

Thompson sampling is a rule for running a multi-armed bandit: for each new visitor, draw one plausible value of every variant's performance from its probability distribution and show the variant with the highest draw. Over time, each variant gets traffic roughly in proportion to the probability that it is the best one.

  • For prices, draw a conversion rate and multiply it by the price, so the rule compares revenue rather than clicks.
  • Uncertain variants still win some draws, which is how the method keeps exploring without a fixed exploration rate.

Thompson sampling decides who sees what by weighted chance. For each price, it keeps a range of values that the true revenue per visitor could plausibly have, given the data so far. When a visitor arrives, it picks one random value from each range and shows the price whose value came out highest. If $39 is probably better than $29 but not certainly, $39 might win seven draws in ten and get about 70% of the next visitors, while a price that is clearly worse almost never wins.

The method is old: it was first proposed in 1933 for allocating patients between treatments in clinical trials, and decades later spread to web experiments and online advertising (Russo et al., A Tutorial on Thompson Sampling).

How to calculate it

When a visitor either buys or doesn't, the standard model gives each price's conversion rate a Beta distribution. Multiplying the drawn conversion rate by the price turns it into revenue per visitor:

Text
for each price:
  conversion draw = random draw from Beta(1 + buyers, 1 + visitors − buyers)
  revenue draw    = price × conversion draw
show the price with the highest revenue draw

Example: after 1,000 visitors each, $29 has 25 buyers and $39 has 21. The average draws are about 2.6% and 2.2% conversion, or $0.75 and $0.86 per visitor. Because the two ranges overlap, $39 wins about 67% of the draws and $29 about 33%, so the next visitors split roughly two to one in favor of $39. As purchases come in, the ranges narrow and the split moves further toward the better price.

Illustration: beliefs about the conversion rates of three prices after 1,500 visitors each. Where the curves overlap, the trailing prices still win some draws.

Why it matters for pricing

Thompson sampling settles the explore-or-exploit question without an exploration rate to tune. Uncertain prices get explored because they sometimes draw high, and the leader gets most of the traffic because it usually does. The same draws also give you the probability that each price is best, which is the number you want when deciding what to charge.

A plain buy-or-not model counts a refunded order as a sale. Leverprice's engine is a Thompson-sampling bandit that models three outcomes per visitor (kept purchase, refunded purchase, no purchase), so refunds lower a price's score, and it recomputes traffic weights every few minutes.

For a step-by-step look at the method on prices, see Thompson sampling for pricing.

Common mistakes

  • Sampling conversion instead of revenue. Comparing raw conversion draws will nearly always favor the cheapest price.
  • Using a strong prior. A prior worth thousands of pseudo-visitors outweighs the real data until thousands of real visitors arrive, while a flat Beta(1, 1) lets the data take over almost immediately.
  • Updating on visitors who haven't had time to buy, which makes the price that got the latest traffic look worse.
  • Reading the traffic split as the size of the effect. A 90% share means a price is very likely the best one and says nothing about how much better it is.

Thompson sampling is one way to run a multi-armed bandit. Its posterior distributions are the same ones used in Bayesian A/B testing, they also give the expected loss of stopping now, and what they rank is revenue per visitor.

Frequently asked questions

Why not always show the current leader?

Early leaders are often lucky. A rule that always shows the leader can lock onto a worse price after one good day and never collect the data to correct itself. Sampling keeps giving the trailing prices a chance in proportion to how likely they are to be best.

How is Thompson sampling different from epsilon-greedy?

Epsilon-greedy shows the leader most of the time and a random variant a fixed share of the time, such as 10%, no matter how clear the evidence is. Thompson sampling explores less as the evidence firms up, so it wastes fewer visitors once one price is clearly ahead.