Multi-armed bandits for pricing: how they work and when they beat A/B tests

Multi-armed bandit pricing explained: explore vs exploit, epsilon-greedy vs UCB vs Thompson sampling, refunds, late purchases, stopping rules, trade-offs.

Short answer

A pricing bandit shows visitors several prices and keeps shifting traffic toward the price that earns the most revenue per visitor. It beats an A/B test when revenue during the test matters: at 300 sales a month, an even three-way split needs about 22 months to confirm a 10% gap. Choose A/B when you need the exact lift.

  • The reward has to be revenue per visitor (price × conversion). A bandit that maximizes conversion drifts to your cheapest price.
  • Thompson sampling gives each price traffic in proportion to its probability of being best, which holds up better than UCB when feedback arrives in batches.
  • Real pricing adds late purchases, refunds and moving markets: Leverprice counts visitors only after 30 minutes, models kept, refunded and no purchase with a Dirichlet, and keeps 5% on every price.
  • In 30 simulated runs at 300 sales a month, the engine decided in a median of 40 days and picked the best price 27 times, but its estimate of the lift at that moment ranged from 4% to over 500%.

Say you sell a $29 plan and want to know whether $19 or $49 would earn more. A classic A/B test splits visitors three ways and waits for proof. At 300 sales a month, proving that $49 earns 10% more per visitor than $29 (95% confidence, 80% power) takes about 22 months of traffic (the sample size math). A multi-armed bandit doesn't wait for proof: every few minutes it moves visitors toward the price that is most likely earning the most, and keeps a trickle on the others in case it's wrong.

This article covers how that works, which algorithm fits pricing, and the parts textbooks skip: purchases that arrive late, refunds, a market that moves under you, and knowing when to stop.

What is a multi-armed bandit, in pricing terms?

A multi-armed bandit is an experiment where every choice earns or loses money while you learn. The name comes from slot machines: several "one-armed bandits" with unknown payouts, and you want to win as much as possible while working out which one pays best.

In a price test, each arm is a price. Pulling an arm means showing that price to one visitor. The reward is what that visitor pays: $0 most of the time, the price when they buy. The cost of learning is called regret: the revenue you would have earned by showing the best price to everyone from day one, minus what you actually earned.

Here is the tension. Suppose the truth, which you can't see, is this: $19 converts 2.6% of visitors ($0.49 per visitor), $29 converts 2.0% ($0.58) and $49 converts 1.3% ($0.64). So $49 is your best price. After 300 visitors it has made 3.9 sales on average, and binomial math gives a 25% chance it has made two or fewer. Two sales from 300 visitors is $0.33 per visitor, well below what $19 and $29 are really earning.

A greedy rule that always shows the best-looking price drops $49 at that moment and never shows it again, so it never learns it was wrong. The opposite rule, an even split until the end, never makes that mistake but keeps paying for the $19 and $29 visitors long after the data has turned against them. A bandit algorithm explores while the evidence is thin and exploits as it firms up.

Why must the reward be revenue per visitor?

Because conversion rate and order value each point at the wrong price. Lower prices convert better, so a bandit that maximizes conversion drifts to the cheapest price you let it test. Higher prices earn more per order, so maximizing order value drifts to the most expensive one. The number that tracks money is revenue per visitor: revenue divided by everyone who saw the price, buyers or not.

Illustration, observed after 1,500 visitors per price:

PriceSalesConversionRevenue per visitor
$19412.73%$0.52
$29302.00%$0.58
$49221.47%$0.72

A conversion-maximizing bandit would pick $19, the price that earns the least per visitor. A revenue-per-visitor bandit picks $49.

Two refinements matter in practice. If refunds are common, count only the purchases that stick. If your costs grow with every sale (payment fees, model costs for an AI product), maximize profit per visitor instead. Leverprice lets you choose the goal: revenue, sales, or profit after provider fees and unit cost.

Epsilon-greedy, UCB or Thompson sampling: which fits pricing?

Thompson sampling. All three classic algorithms balance exploring and exploiting; they differ in how they decide to explore, and that matters when rewards are rare, delayed and measured in dollars.

Epsilon-greedy

Epsilon-greedy shows the best-looking price most of the time and a random price with a small probability ε, say 10%. It is easy to write and easy to explain. Its exploration is blind: a price that is clearly losing gets the same random share as one that is nearly tied, forever, so with a fixed ε the lost revenue keeps growing in a straight line (Auer, Cesa-Bianchi and Fischer, 2002). Its greedy part trusts noisy early averages, which is exactly the $49 trap above.

UCB

UCB (upper confidence bound) adds an optimism bonus to each price's average and picks the highest total. UCB1 uses average + sqrt(2 ln t / n), where t counts all visitors and n the visitors this price has seen, and it has a proven logarithmic bound on regret for rewards between 0 and 1 (same paper). Two things hurt it in pricing. It is deterministic, so if you recompute every five minutes, every visitor in those five minutes gets the same price. And its bonus ignores how rare purchases are: with rewards scaled by a $49 maximum, after 10,000 visitors per price the bonus is still worth about $2.22 per visitor, while in the example above the three prices differ by 6 to 15 cents. UCB1 keeps splitting traffic almost evenly for a very long time.

Thompson sampling

Thompson sampling keeps a probability distribution over each price's conversion rate, draws one plausible value per price, multiplies by the price, and shows the winner of that draw. Each price ends up with traffic in proportion to its probability of being the best, which is why the method is also called probability matching (Chapelle and Li, 2011). William R. Thompson proposed it in 1933 for clinical trials, and for yes-or-no rewards its regret matches the best rate any algorithm can achieve in the long run (Kaufmann, Korda and Munos, 2012).

Why Thompson sampling suits pricing

Four properties make it the default choice for price tests:

  1. Feedback in pricing always arrives late and in batches: checkout takes minutes, webhooks follow, and the allocation job runs on a schedule. Randomized allocation copes with that. In Chapelle and Li's simulation, UCB's regret was 2.65 times Thompson sampling's with instant feedback and 3.82 times with feedback every 1,000 steps. Scott (2010) makes the same point about greedy rules, which play one arm for an entire update cycle.
  2. Revenue fits in one line: sample a conversion rate, multiply by the price. Refunds and profit fit by swapping the Beta distribution for a Dirichlet, shown below.
  3. The same draws give the probability that each price is best and the expected loss of picking it, in dollars per visitor. Those are the two numbers you need to decide when to stop.
  4. There is no ε to tune.
Epsilon-greedyUCB1Thompson sampling
How it exploresRandom share ε, foreverOptimism bonusDraws from the posterior
Batched updatesGreedy share goes to one price per batchWhole batch goes to one priceFine, allocation is random
Reports P(best) and expected lossNoNoYes
Settings to chooseεReward scalingPrior

How does Thompson sampling choose a price?

It turns what you have seen into a belief about each conversion rate, then samples from those beliefs. With a uniform starting belief, the belief about a conversion rate after s sales from n visitors is a Beta(1 + s, 1 + n − s) distribution. For the illustration above it looks like this:

Illustration: 41, 30 and 22 sales from 1,500 visitors per price. $49 converts least but earns the most per visitor.

The $49 curve sits furthest left because it converts least. Multiply each curve by its price, though, and $49 comes out ahead in most draws: across 4 million draws it wins 75.6% of the time, $29 wins 18.9% and $19 wins 5.4%. Pure Thompson sampling would send the next visitors in roughly those shares, and as sales accumulate the curves narrow and the shares follow. The Thompson sampling walkthrough builds this from scratch in TypeScript and Python.

In production you don't need a fresh draw per visitor. Our engine runs every 5 minutes: it takes 4,000 posterior draws per price, counts how often each price wins (that is its probability of being best), moves the current split halfway toward those probabilities, and keeps every price at 5% or more. Weights change only when some price would move by at least one percentage point, so traffic doesn't twitch on noise. Before any of that, every price needs 100 visitors and the test needs 10 purchases; until then the split stays even.

Here is what that looks like over a whole test, in one run from our simulation of the production engine's allocation code (updated four times a day in the simulation): $19, $29 and $49 with $29 as the current price, 300 sales a month (500 visitors a day), $49 earning 10% more per visitor than $29 and $19 earning 15% less.

Simulation of the production engine at 300 sales a month: the run whose decision day was closest to the median of 40 runs.

The engine moved 74% of traffic to $49 by the end of day 2, let $19 climb back to 43% around day 19 while $49's sales ran cold, and then settled: on day 40, $49 had 89% of traffic and the other two sat near their 5% floors. Over those 40 days, 62% of visitors saw $49, against 33% under an even split.

What makes pricing harder than a textbook bandit?

Textbook bandits assume each reward arrives the moment you pull the arm and that the payouts never change. Pricing violates both, and adds refunds and returning visitors on top.

Purchases arrive late

A visitor who opened your pricing page two minutes ago hasn't had time to buy. Count them, and every price looks worse in proportion to how much fresh traffic it just received. In a bandit, the price gaining traffic receives the most fresh visitors, so it gets penalized hardest exactly when it is winning, and traffic swings back and forth.

Our engine leaves out visitors whose first visit was less than 30 minutes ago, together with their purchases. Purchases that arrive later are still credited to the price that visitor saw. If your buyers usually take days (a trial that converts at the end, a team that needs sign-off), the window removes the same-session bias but not the multi-day one. That leftover bias pushes against whichever price is gaining traffic, so it slows the test down; it doesn't steer traffic toward a worse price.

Refunds turn a sale back into nothing

A purchase refunded next week was never revenue. Count it, and a price that attracts buyers who change their minds looks better than it is. So the engine records one of three outcomes for every visitor: kept purchase, refunded purchase, no purchase. Their probabilities get a Dirichlet posterior, the three-outcome version of the Beta, from a uniform prior:

Text
(pKept, pRefunded, pNone) ~ Dirichlet(1 + kept, 1 + refunded, 1 + visitors - kept - refunded)
value per visitor = pKept × vKept + pRefunded × vRefunded

Under the revenue goal, a refunded purchase is worth zero and a kept one is worth the average amount actually kept (coupons and partial refunds included, pulled toward the list price by 5 pseudo-purchases so a few discounted orders can't swing it). Under the profit goal, the engine counts a refund as a loss, because it assumes the payment fee and the unit cost are not recovered. That negative value is why refunds get their own outcome instead of being folded into "no purchase".

Your market moves while you test

The math assumes each price's conversion rate holds still. Real ones drift: weekends convert differently from weekdays, and a launch brings a different crowd. Two problems follow.

First, a bandit changes its split over time, so the prices see different mixes of good and bad days. Microsoft's experimentation team showed how that bites even in plain A/B tests whose split changed midway: a treatment that was better on Friday and better on Saturday came out worse when the two days were pooled (Crook et al., 2009). Second, if the best price really changes, a bandit that has stopped exploring is slow to notice. Russo et al. suggest ignoring or discounting old data, which keeps the bandit exploring forever and so never lets it settle on an answer.

Our engine pools every visitor past the 30-minute mark since the test began, keeps at least 5% of traffic on each price, and moves the split only halfway toward its new target at each update, which limits how far one odd day can drag it. The practical rules are simple: start on an ordinary week, judge results in whole weeks, and run a campaign or holiday sale as its own test.

Every price keeps a 5% floor

A price that gets no traffic can never prove it was underrated. The floor costs a little: with three prices the leader can take at most 90%, which in the simulated example keeps you about 1.6% below the best price's revenue per visitor while the test runs. You stop paying it when you lock in the winner.

A visitor has to keep their price

If someone sees $29 on Monday and $49 on Tuesday, you have an annoyed customer and corrupted data. The split decides what new visitors get; anyone already assigned keeps their price. This is called sticky assignment, and we do it with a signed first-party cookie and, for signed-in users, a hashed user ID.

When should a pricing bandit stop?

When one of two rules fires. Our engine checks both at every update after warmup:

  1. One price is at least 95% likely to be the best. The engine calls this a clear winner, and Scott's batch simulations stopped at the same 0.95 threshold.
  2. Picking the current leader is expected to cost less than 0.5% of its revenue per visitor. The engine calls the prices practically equivalent.

Expected loss is the average, over the posterior, of how much revenue per visitor you give up by picking the leader when another price is actually better: E[max over prices − leader]. It is in dollars, so you can read it as the price of being wrong. Bayesian A/B testing calls the cutoff a "threshold of caring" (Stucchio, 2014).

In our simulations the second rule ends most tests. In the chart's run, $49 was only 91% likely to be best on day 40, but its expected loss was $0.0027 per visitor, 0.49% of its estimated $0.56, so the engine called it. Across the 30 simulated runs at 300 sales a month, 7 ended as clear winners and 23 as practically equivalent. Once a winner is called, you lock it in with one click and it is live for everyone in about a second.

What do you give up with a bandit?

A precise estimate of how much better the winner is. A bandit stops sending traffic to prices it believes are worse, so it learns little more about them. Their estimates stay wide, and they tend to sit too low, because a price that looked unlucky early got fewer chances to recover (Nie et al., 2018). Scott's simulations found credible intervals for losing arms much wider under probability matching than under an even split.

Our simulation shows the size of it. Across 30 runs at 300 sales a month, where $49 truly earns 10% more per visitor than $29, the engine picked $49 in 27 runs and $29 in 3, and never $19. But in the 27 correct runs, its estimate of the lift over $29 at the moment it stopped ranged from 4% to more than 500%, with a median of 19%. In the chart's run it put $29 at $0.41 per visitor (90% interval $0.30 to $0.54) when the truth was $0.58.

A bandit answers "which price?" well and "by how much?" poorly. If you need the number itself for a board deck or a forecast, run a fixed split, or follow the bandit with one. The bandit vs A/B test comparison works through what each option costs in visitors and dollars.

When does a bandit beat an A/B test for pricing?

When the money you earn during the test matters more than the precision of the final number.

SituationBetter choice
You want the most revenue while the test runsBandit
You are testing three or more pricesBandit
You want a test that can keep running safelyBandit
You need the exact lift for a board deck or forecastFixed A/B split
Churn or retention by price matters more than the first purchaseFixed A/B split
Very little traffic, and you want one simple comparisonFixed A/B split, two prices

Leverprice runs the bandit described here on prices your own server computes, with revenue verified by Stripe, Polar, Lemon Squeezy or Dodo Payments webhooks (how Leverprice tests prices). Whichever tool you use, check these before you start:

  1. The goal is revenue or profit per visitor, never conversion rate.
  2. Refunds count as their own outcome.
  3. Visitors too fresh to have bought are left out.
  4. Every price keeps a minimum share of traffic.
  5. Returning visitors keep the price they saw.
  6. The stop rule is fixed in advance: 95% probability of being best, or expected loss under 0.5%.
  7. Nobody quotes the bandit's lift estimate as precise.

Frequently asked questions

Is a pricing bandit the same as dynamic pricing?

No. A bandit tests a fixed set of prices you chose, gives each visitor one of them, and ends with a single price for everyone. Dynamic pricing keeps adjusting prices over time, often with demand, with no final answer.

Will a returning visitor see a different price as traffic shifts?

Not if assignment is sticky. The split only decides the price for new visitors; anyone already assigned keeps their price, including signed-in users on another device.

How long does a pricing bandit take?

It depends on your sales volume and the size of the gap. In our simulation of the production engine with a 10% gap, the median was 113 days at 100 sales a month, 40 days at 300 and 14 days at 1,000 (how long a price test takes).

Can a bandit pick the wrong price?

Yes. In the simulation above it chose $29 over $49 in 3 of 30 runs, and $29 earns about 9% less per visitor. Two of those three calls came in the first four days, on 19 and 39 sales in total, so treat a very early call with suspicion and let the test run at least a full week before you lock it in.

How many prices should I test at once?

Each extra price needs its own evidence before the bandit can rule it out, so more prices mean a longer test. Start with your current price plus one lower and one higher, and add more only when traffic allows.