Multi-armed bandit: definition, formula and examples

A multi-armed bandit moves traffic toward the better variant while a test runs, so fewer visitors see losing prices. How it works, the math and the tradeoffs.

Definition

A multi-armed bandit is a testing method that shifts traffic toward the best-performing variant while the test is still running, instead of holding a fixed split until the end. It accepts a less precise estimate of the losing variants in exchange for sending fewer visitors to them, which suits price tests run on real buyers.

  • Every visitor sent to a worse price costs the gap in revenue per visitor, and a bandit tries to keep that cost low.
  • A bandit still needs a floor of exploration and a delay for outcomes that take time, such as purchases.

A multi-armed bandit is a test that learns as it goes. Instead of splitting visitors evenly between $19, $29 and $49 for a month and only then picking a winner, it starts even, watches who buys, and keeps sending more visitors to whichever price is earning the most. By the second week the leading price might get 70% of traffic and the weakest one 10%.

The name comes from slot machines, once nicknamed one-armed bandits: a gambler facing a row of them has to decide which arm to keep pulling while still learning which machine pays best (background).

How to calculate it

A bandit is a rule for splitting traffic. The cost it tries to keep small is called regret: the revenue lost by showing visitors anything other than the best price.

Text
regret = Σ over visitors (RPV of best price − RPV of price shown)

Example: say $49 earns $0.64 per visitor, $29 earns $0.58 and $19 earns $0.49. Over 9,000 visitors, an even split sends 3,000 to each losing price and gives up 3,000 × $0.06 + 3,000 × $0.15 = $630. A bandit that splits the first 3,000 visitors evenly ($210 of regret), then sends 80% of the next 6,000 to $49 and 10% to each of the others ($126), gives up $336, about half as much.

How fast traffic moves depends on the rule. Thompson sampling, a common choice for prices, gives each price a share of traffic roughly equal to the probability that it is the best.

Our simulation of the production engine at 300 sales a month, with $49 earning 10% more per visitor than $29: the even split never moves, while the engine sends most visitors to $49.

Why it matters for pricing

In a price test the arms are prices and every pull is a real buyer, so a losing price costs money for as long as it gets traffic. An A/B test pays that cost on purpose to measure every price precisely. A bandit stops paying it once the evidence points one way. Leverprice runs a Thompson-sampling bandit that recomputes traffic weights every few minutes, keeps at least 5% of traffic on every price and counts only visitors older than 30 minutes, so fresh traffic that hasn't had time to buy doesn't drag a price down.

The cost is precision. Losing prices get fewer visitors, so their numbers stay noisy, and a bandit is the wrong tool if you need an exact estimate of how much worse $19 is. It is the right one when the question is which price to charge.

Common mistakes

  • Reading the traffic share as the result. A price with 70% of traffic is the current favorite, and the size of its advantage still has to come from the revenue numbers.
  • Reacting to purchases that haven't happened yet. Visitors from the last few minutes have had no time to buy, so counting them punishes whichever price got the newest traffic.
  • Turning exploration off. With no minimum share, a price that had an unlucky first day can lose all its traffic and never get the chance to recover.
  • Changing the page or the traffic mix mid-test, which the bandit reads as a change in how the prices perform.

Expected loss is how a bandit decides it can stop. The fixed-split alternatives are a Bayesian A/B test and a classic test judged by statistical significance, both scored on revenue per visitor when prices are involved.

Frequently asked questions

Is a bandit better than an A/B test for pricing?

If your goal is to end up charging the best price while losing as little revenue as possible along the way, a bandit fits better. If you need a precise estimate of every price's effect, a fixed split does. The tradeoffs are laid out in bandit vs A/B test for pricing.

Does a bandit need less traffic?

Not to tell two close prices apart. The weaker price gets fewer visitors, so its estimate firms up more slowly. What a bandit saves is revenue, because most visitors see the leading price while the evidence builds.