Original title: Robust A/B Decisions
Max H. Farrell, Malika Korganbekova, Sanjog Misra · September 2026
Companies use A/B tests to compare two versions of a product, like an ad. Normally, they pick the new version if a math test shows a statistically significant gain. But this traditional method answers the wrong question: it only checks the past test data, not the real money the company will make in the future. To fix this, researchers created a new decision rule that plans for uncertainty. Instead of assuming the future will match the test exactly, the rule looks at possible outcomes near the experimental results. It then lowers the estimated value of each option to protect against bad surprises. The system needs only standard test data and one simple setting for how much to trust the experiment. When tested on real data from 552 digital advertising experiments, this conservative method substantially reduced regret compared to the old approach.
Imagine tasting a spoonful of soup to decide whether to serve it to guests. If you only taste one spoonful, you might get a misleadingly good or bad sample, and then either serve bad soup or waste good soup. The usual approach—taste once and decide—is like the standard A/B test, which ignores how much the soup can vary from one spoonful to another. Here, the researchers create a rule that, based on that single taste and your confidence in the sample, tells you whether to serve the whole pot, accounting for the possibility that other spoonfuls might not taste as good.
Companies running A/B tests often deploy treatments that look statistically significant but fail to deliver real economic value. This new rule considers future performance and penalizes uncertainty, leading to fewer costly mistakes. In tests with 552 real advertising experiments, it significantly reduced regret compared to standard methods.
The evaluation is limited to 552 advertising experiments from a single anonymous US online platform, so results may not generalize to other domains or platforms. The rule requires specifying an interpretable trust parameter, but the abstract does not indicate how to calibrate it in practice. The framework assumes the future deployment environment is similar to the experimental sample, which may not always hold.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Robust A/B Decisions — Max H. Farrell, Malika Korganbekova, Sanjog Misra
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.