Should you never A/B test your pricing? Paddle's 3 arguments, checked against the numbers

Paddle says A/B testing your pricing is never a good idea. We build a price testing tool, so here is a fair check of each argument, with simulation numbers.

Short answer

Paddle's case against price A/B tests is right about small businesses, anchoring and what a test can't tell you. It is outdated on the rest: adaptive tests need far less traffic than fixed splits, sticky server-side assignment handles most fairness concerns, and a survey plus a test beats either alone.

  • Paddle is right that a classic fixed test is slow: 663 days for a 10% better price at 300 sales a month in our model.
  • An adaptive test found the same price in a median of 40 days in our simulations, because it stops when the remaining difference stops mattering.
  • Surveys tell you what people say they'd pay; tests tell you what they pay. Use the survey to pick the range, then test inside it.

In September 2022 Paddle published Price testing: 3 reasons to never A/B test your pricing strategy. Its verdict is blunt: "A/B testing your pricing strategy is never a good idea." Instead, it recommends surveying customers about their willingness to pay. The post still ranks for searches about price testing, so plenty of founders read it before deciding what to do with their price.

We build Leverprice, a price testing tool, so we are not neutral. That is exactly why it's worth going through each argument carefully, saying where Paddle is right, and showing numbers where things have changed. Some of their points hold up well.

Argument 1: "You rarely get statistical significance"

Paddle's first argument is that each price needs a large number of buyers before a classic test can call a winner, so small businesses can't test at all, and even large ones wait a month or more.

Where Paddle is right. A fixed, even-split test on revenue per visitor needs a lot of traffic. We calculated it for a store at $29 with a 2% conversion rate, testing a $49 price that earns 10% more per visitor, at 95% confidence and 80% power: about 110,000 visitors per price. At 300 sales a month with three prices on an even split, that is 663 days. At 100 sales a month it is over five years. For a young product, a textbook A/B test of prices is not realistic, and Paddle says so plainly.

What has changed. A fixed test is built to measure the lift precisely. Most founders don't need a precise lift, they need to know which price to run. Adaptive engines answer that question with a different stopping rule: keep moving traffic toward the leader, and stop when picking the current leader costs less than a tiny expected loss. We ran 960 simulations of our production engine on the same kind of store. At 300 sales a month it found the 10% better price in a median of 40 days, and across 870 decisions it never settled on the worst price (full study).

That isn't a free lunch. You get the right price faster, not a defensible estimate of the lift. And below roughly 50 sales a month, even adaptive tests mostly need a big gap between prices to finish within a year.

On "price sensitivity shifts within days". The post says evidence suggests this, without citing any. If it were true, it would also undermine surveys, which take longer to run than many tests. In practice, sensitivity does move with seasons, campaigns and launches, which is a reason to keep a test running through ordinary weeks, not a reason never to test.

Argument 2: "Testing anchors customers and upsets them"

The second argument has two parts. A visitor who saw a low price will resist a higher one later, and showing different visitors different prices is an ethical gray area: people clear cookies, compare notes with colleagues and complain.

Where Paddle is right. This is the most serious objection, and anyone running a price test should take it seriously. Anchoring is real. A customer who finds out a colleague paid less will be unhappy, whatever the reason. And a test that shows one price on the page and charges another at checkout is not a test, it's a bait-and-switch.

What you can do about it. Most of the risk comes from how the test is run, not from testing itself:

  • Keep each person on one price. Assign the price on the server and keep it in a signed first-party cookie, so every return visit shows the same price. If a price must follow someone across devices, store the price they were shown on their account. Clearing cookies can reshuffle an anonymous visitor, which is one more reason to keep test prices within a range you'd defend.
  • Test on new visitors only. Existing customers keep what they have. If you want to change their price, that's a separate decision with notice and grandfathering (how to raise prices without losing customers).
  • Charge exactly what you showed. The checkout amount must come from the same assignment as the page.
  • Keep the prices close to a range you'd defend in public. If you can't explain to a customer why $39 and $49 were both reasonable, don't test them.
  • Have a support answer ready. "We test prices on new visitors to find a fair one; here's what we can do for you" handles most cases. Some teams simply honor the lower price when asked.

On the ethics: random assignment that doesn't use anyone's personal data is a different practice from personalized pricing, and that distinction matters legally as well. We wrote a separate guide on whether price testing is legal in the US, EU and UK.

Argument 3: "A/B testing is relative, not absolute"

Paddle's third point is that a test only says which of the prices you tried did better. It doesn't reveal what customers would actually pay, so the winner may still be far from the best price, and you can't keep retesting forever.

Where Paddle is right. Completely. A test can only pick among the prices you put in it. If you test $19, $24 and $29 when the best price is $59, the test will happily crown $29.

What has changed. The answer isn't "survey instead of test", it's "survey, then test". A Van Westendorp survey gives you a range people find acceptable. A test inside that range tells you which price earns the most from people who actually reach your checkout. The two answer different questions: what people say they'd pay, and what they do pay. Surveys are cheap and fast but suffer from the gap between stated and revealed preferences. Tests are slower but measure money. And "continuously retesting isn't practical" was true when every test was a manual project; an engine that keeps running and shifts traffic as behavior changes does the retesting for you.

Where this leaves you

Paddle's argumentHolds whenWhat changed since 2022What to do
Significance is rarely possibleYou run a fixed even split or sell fewer than about 50 times a monthAdaptive tests stop on expected loss and need far less trafficTest bigger gaps, use an adaptive engine, survey first if volume is tiny
Testing anchors and upsets customersPrices aren't sticky, existing customers are included, or checkout differs from the pageServer-side sticky assignment tied to accounts is easy to set upNew visitors only, one price per person, charge what you show, prepare support
Tests are relative, not absoluteYou test a range that misses the best priceNothing, this is trueUse a survey to pick the range, then test inside it

Our position, after reading the post twice: don't run a classic A/B test on prices if you sell a few dozen times a month, and don't run any price test carelessly. Do run one when you have steady sales, a range you'd defend in public and a way to keep each person on one price. If you want to see what that looks like on your own product, here is how Leverprice does it: your current price stays the control, prices are decided on your server, and only purchases your payment provider confirms are counted.

Frequently asked questions

Is it a good idea to A/B test pricing?

It is if you have steady sales, test new visitors only, keep each person on one price and charge exactly what you show. It is a bad idea with very low volume, on existing customers without notice, or with prices you couldn't defend if a customer compared notes.

Are pricing surveys better than price tests?

They answer different questions. Surveys estimate what people say they'd pay and are good for choosing a range. Tests measure what visitors actually pay. Using a survey to pick the range and a test to pick the price inside it is stronger than either alone.

How much traffic do I need to test prices?

With a 2% conversion rate, a fixed test needs roughly 110,000 visitors per price to detect a 10% difference in revenue per visitor. Adaptive tests need far less to pick a price; in our simulations that took a median of 40 days at 300 sales a month.

What if a customer finds out someone paid less?

Explain that you test prices on new visitors to find a fair one, and decide in advance whether you'll honor the lower price on request. Many teams do. Keeping test prices within a range you'd defend publicly makes this conversation easy.