# Bandit or A/B test for pricing: which one costs you less money?

Source: https://leverprice.com/blog/bandit-vs-ab-test-pricing
Updated: 2026-11-17

> At 300 sales a month, an even split of $19, $29 and $49 needs about 22 months to confirm a 10% gap and sends 110,451 visitors to the worst price. In simulation, our bandit decided in a median of 40 days. Pick A/B when you need the exact lift, the bandit when revenue during the test matters.

You sell at $29 and want to test $19 and $49, at 300 sales a month. To prove with a standard fixed-horizon test that $49 earns 10% more per visitor than $29, an even three-way split needs about 331,000 visitors: 22 months of your traffic, a third of it on your worst price. In our simulation of the Leverprice engine, a Thompson-sampling bandit made the call in a median of 40 days. The bandit pays for that speed in precision: it is good at naming the winner and poor at measuring the margin.

Below is the full arithmetic for both approaches on the same market, what each one costs in visitors and dollars, and a table for deciding which one fits your test. For how the bandit itself works, see [multi-armed bandits for pricing](https://leverprice.com/blog/multi-armed-bandit-pricing).

## What exactly are you trading?

Revenue during the test against precision at the end. A fixed even split measures every price equally well because every price gets a third of the traffic until the end. That includes the worst price, which keeps getting a third of your visitors long after the data has turned against it. A bandit moves traffic toward the leader as evidence comes in, so it earns more while the test runs, but the prices it abandons get few visitors and stay poorly measured.

Both sides of this trade show up in published simulations. [Scott (2010)](https://doi.org/10.1002/asmb.874) compared an equal split with randomized probability matching (Thompson sampling) on ten arms with batch updates: the equal split's regret was more than an order of magnitude larger, while it produced more accurate estimates of the losing arms, at a much higher price. For pricing specifically, [Misra, Schwartz and Abernethy (2019)](https://doi.org/10.1287/mksc.2018.1129) built a bandit price-experimentation policy and, in a simulation calibrated to a real pricing field experiment, estimated it could raise profits by 43% during the month of testing and 4% annually.

## Worked example: three prices, 300 sales a month

This is the same market as the simulation in the chart further down:

| Price | Conversion | Revenue per visitor | vs $29 |
|---|---:|---:|---:|
| $19 | 2.59% | $0.493 | −15% |
| $29 (current) | 2.00% | $0.580 | |
| $49 | 1.30% | $0.638 | +10% |

At a 2% conversion rate, 300 sales a month means 15,000 visitors a month, or 500 a day. Every purchase is at list price and nobody refunds, to keep the arithmetic clean.

### How many visitors does an even split need?

110,451 per price, or 331,353 in total. Each visitor brings in either $0 or the price, so revenue per visitor behaves like a conversion rate scaled by the price: its mean is price × p and its variance is price² × p × (1 − p). The standard two-proportion sample size formula applies with every term scaled that way. Per price, for a two-sided test at 95% confidence and 80% power, comparing $49 with $29:

```text
n = (z_0.975 + z_0.80)^2 × (var_29 + var_49) / (mean_49 - mean_29)^2
```

The arithmetic:

```text
z_0.975 + z_0.80  = 1.959964 + 0.841621 = 2.801585, squared = 7.848879
p_49              = 0.638 / 49 = 0.013020408
var_29            = 29^2 × 0.02 × 0.98 = 16.4836
var_49            = 49^2 × 0.013020408 × 0.986979592 = 30.8550
mean_49 - mean_29 = 0.638 - 0.580 = 0.058, squared = 0.003364

n = 7.848879 × (16.4836 + 30.8550) / 0.003364 = 110,450.3, rounded up to 110,451 per price
```

Three prices at 110,451 each is 331,353 visitors. At 500 a day that is 663 days, about 22 months, and the worst price, $19, gets 110,451 of them. This compares $49 with $29 only; correcting for comparisons among all three prices would make it longer. The 10% gap is your [minimum detectable effect](https://leverprice.com/glossary/minimum-detectable-effect), and the [price test sample size calculator](https://leverprice.com/tools/price-test-sample-size-calculator) sizes a test for your own numbers.

Two checks on the formula. With equal variances it matches the familiar 16σ²/δ² rule of thumb ([Evan Miller](https://www.evanmiller.org/how-not-to-run-an-ab-test.html)), since 7.85 × 2 ≈ 16. And it has to run on revenue per visitor, not conversion. $49 converts 35% worse than $29, a gap an even split detects with about 5,200 visitors per price, so a conversion-rate test would declare your best price the loser.

### What does the even split cost?

Every visitor sent to $19 earns $0.145 less than if they had seen $49, and every visitor sent to $29 earns $0.058 less. Over the full test:

```text
$19: 110,451 × $0.145 = $16,015
$29: 110,451 × $0.058 = $6,406
total: about $22,400 below showing everyone $49 for 22 months
```

In return you get a precise answer. At the end, the 95% confidence interval on the difference between $49 and $29 is ±$0.041 per visitor, so you can report the lift as 10% ± 7 points.

### What does the bandit do with the same traffic?

We ran the production engine's allocation code on this exact market 30 times with different random seeds, with the default 5% floor per price. The simulation updates the split four times a day; the live engine does it every 5 minutes. The chart shows the run whose decision day was closest to the median:

*Figure: Two stacked area charts of daily traffic share for $19, $29 and $49 over 40 days: the even split stays at one third each, while the engine moves most traffic to $49. Simulation of the production engine at 300 sales a month, $49 earning 10% more per visitor than $29. Left: even split. Right: the engine's daily split until its decision on day 40.*

In this run the engine had moved 55% of traffic to $49 by the end of day 1 and 74% by the end of day 2. Then $49's sales ran cold for a few weeks (about 1.05% of visitors against a true 1.30%), $19 drew level, and the engine hedged: $19 climbed back to 43% of the traffic around day 19 before the evidence turned. By day 40, $49 had 89% of the traffic and the engine called it, with an expected loss under 0.5% of revenue per visitor. Over those 40 days $19 got 4,983 visitors, $29 got 2,444 and $49 got 12,323.

That run spent more on $19 than most. Across all 30 runs:

1. The engine reached a decision in every run, with a median of 40 days (fastest 1 day, slowest 251).
2. It picked $49 in 27 runs and $29 in 3, and never picked $19.
3. The median number of visitors sent to $19 before the decision was 2,408, and the median share was 12%.
4. The median shortfall against showing everyone $49 from the start was $852.

### How do the options compare side by side?

The middle column is the tempting shortcut: run an even split for a few weeks and pick whichever price looks best.

| | Even split, full test | Even split, stopped at day 40 | Bandit, median of 30 runs |
|---|---:|---:|---:|
| Days | 663 | 40 | 40 |
| Visitors on $19 | 110,451 | 6,667 | 2,408 |
| Revenue below always showing $49 | about $22,400 | $1,353 | $852 |
| Chance of ending on $49 | 80% power to detect the gap | 75% | 27 of 30 runs |
| Precision of the lift | ±7 points | ±28 points | estimates from 4% to 526% |

The day-40 even split numbers are our calculation from the same conversion rates: 20,000 visitors split three ways, then the price with the highest observed revenue per visitor wins. The full even split is the only option that delivers a precise lift, and it costs 26 times more revenue than the median bandit run. Stopping the split early saves the time but ends on the wrong price one time in four. The bandit costs the least and ends on the right price most often, but its number for the lift is the least trustworthy of the three.

## What do you give up with the bandit?

Mostly the lift. In the 27 runs that picked $49, the engine's estimate of its lift over $29 at the moment it stopped ranged from 4% to 526%, with a median of 19%. The truth was 10%. In the chart's run it put $29 at $0.41 per visitor when the truth was $0.58: $29 had a slow start, lost its traffic, and never got enough visitors to recover. That downward pull on abandoned options is a known property of adaptively collected data ([Nie et al., 2018](https://arxiv.org/abs/1708.01977)). [Russo et al.](https://arxiv.org/abs/1707.02038) make the same point about Thompson sampling in general: once it is fairly sure which option is best, it shows that option almost every time and learns about the others very slowly.

You also give up the frequentist error guarantee. The bandit's stop rule asks one question: is picking the leader cheap if it turns out wrong? Statistical significance doesn't enter into it. In the 30 runs it picked $29, which earns 9% less than $49, three times, and two of those calls came in the first four days on 19 and 39 sales. A very early call deserves another week of data before you lock it in.

The A/B side has its own trap. If you watch a fixed-horizon test daily and stop at the first significant result, you no longer have a fixed-horizon test: in [Evan Miller's example](https://www.evanmiller.org/how-not-to-run-an-ab-test.html), repeated significance testing turns a nominal 5% false-positive rate into 26.1%. A fixed split only gives you its precision if you commit to the sample size and wait.

## Which should you choose?

Start from what matters most to you in this test:

| If this matters most | Choose | Why |
|---|---|---|
| An exact effect size for a board deck or forecast | A/B test, fixed split | Equal traffic gives every price the same, known precision |
| Long-term churn or retention by price | A/B test, fixed split | You need large, comparable cohorts on each price to follow for months; a bandit leaves losing prices with small cohorts that joined at different times |
| Very few visitors, and you want a simple read | A/B test, two prices | Two prices need two thirds of the traffic of three, and one comparison is easy to explain |
| Revenue while the test runs | Bandit | It shifts traffic to the leader as evidence comes in |
| Three or more prices | Bandit | It stops paying for clear losers early instead of at the end |
| A test you can leave running | Bandit | Leaving it on costs only the 5% floor on each losing price |

Leverprice runs both: the Free plan runs a fixed even split, and Pro runs the Thompson-sampling bandit described here with a 5% floor and two stop rules ([plans and pricing](https://leverprice.com/pricing)). The [Thompson sampling walkthrough](https://leverprice.com/blog/thompson-sampling-pricing) shows the same method in code you can run yourself.

## Can you get the precision and the revenue?

Partly, by choosing what to measure precisely. Run the bandit to find the winner, then, if you need the number, run a fixed 50/50 split between the winner and the runner-up. You skip paying for the worst price, and two prices need 220,902 visitors in this example instead of 331,353.

The bigger lever is the size of the gap you test for. Sample size grows with one over the gap squared, so testing prices further apart shortens both methods. With a 20% gap in the same setup, the even split needs 176 days instead of 663, and the bandit's median falls from 40 days to 23 ([how long a price test takes](https://leverprice.com/blog/how-long-should-a-price-test-run)).

A decision rule that covers most cases: if you will put the lift in a deck, run a fixed split and budget for the visitors it needs before you start. Otherwise run the bandit, let it decide, give an early call one more week, and lock in the winner.

## Frequently asked questions

### Isn't a bandit just an A/B test that peeks?

No. A fixed-horizon test's error rates assume a single look at the end, while a bandit is built to be checked continuously and moves traffic as it goes. It has its own error rate instead: 3 wrong picks in 30 simulated runs in this example.

### How long would a bandit take with my traffic?

Mostly a function of sales per month and the size of the gap. For a 10% gap, our simulation of the production engine needed a median of 113 days at 100 sales a month, 40 days at 300 and 14 days at 1,000.

### Does a two-price A/B test finish faster?

Yes. Two prices at 110,451 visitors each is 220,902 visitors, or 442 days at 500 a day instead of 663. At this traffic level the size of the gap matters more than the number of prices.

### Why not compare conversion rates instead of revenue per visitor?

Because a higher price usually converts worse. In this example $49 converts 35% worse than $29 and still earns 10% more per visitor, so a conversion test would call the best price the loser.

### What if I only get a few dozen sales a month?

Then neither method will give a fast answer for a 10% gap. Test prices further apart, test two at a time, and read [price testing with low traffic](https://leverprice.com/blog/price-testing-low-traffic) for what still works.
