A/B testing upsell offers without fooling yourself
An A/B test shows two or more versions of an offer to comparable customers and compares what they bring in. On upsells it is the fastest way to find the right product and the right discount, as long as you avoid the traps that make a test say whatever you were hoping for.
Change one thing at a time
Test the product, or the discount, or the wording, but not all three at once. If version B wins with a different product and a bigger discount, you wonât know which of the two did the work, and you canât reuse the lesson.
Compare revenue per view, not acceptance rate
The acceptance rate says how often an offer is taken. It doesnât say how much it earns. An item at $5 accepted by 10% of customers earns $0.50 per view. An item at $15 accepted by 5% earns $0.75. The second offer loses on acceptance and wins on revenue.
Revenue per view counts every view, including the refusals at zero. It answers the question that matters: which offer brings more money each time it is shown?
Wait for enough views
Early results swing wildly. After twenty views, one acceptance more or less moves the rate by five points. A test needs a decent number of views on each version before the gap means anything, and how long that takes depends on how often the offer is shown, not on the calendar. A store with thirty orders a week will wait weeks, not days.
Donât stop at the first good-looking gap
Checking a test every day and stopping as soon as one version leads is the most common way to pick a false winner. Differences appear and disappear by chance; if you stop at a high point, you have recorded the chance, not the result. Decide in advance what you need to see, and let the test reach it.
More versions need more proof
With three versions, the best of the three will often be ahead by luck alone, because you are picking the highest of three noisy numbers. The bar has to be higher than with two. Tools that take this seriously raise the required confidence when you add a version.
What to do with the result
- Keep the winner, then put it against a new challenger. Testing never really ends.
- If nothing separates after a long run, the versions are close: keep the one with the better margin or the simpler stock.
- Re-test when the season or the catalog changes. A winner in December may not win in June.
How FlatUpsell reads a test
In FlatUpsell, a post-purchase rule can carry up to three products. The app compares them on revenue per view and computes nothing below twenty views per version, because under that an approximation would be a polite lie.
It says a version wins only at 95% confidence with two versions, and 97.5% with three, which is the same 95% shared across the extra comparison. Below that it says a version leads. The threshold is written on the screen, so you can decide earlier if you accept the risk.