Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 34

A/B test: how many users should be tested?

A population of \(N\) users will be shown version A or B. First \(n\) users per arm are used to test, the rest receive the version that looks better (test and roll). With Gaussian priors \(\mu_j\sim\mathcal N(m,\tau^2)\) on the mean reward of each arm and noise \(\sigma\), the expected total reward is \(Nm+(N-2n)\,\tau^2/(\sqrt\pi\sqrt{\tau^2+\sigma^2/n})\). Testing more makes the choice more reliable but uses up users; the best size \(n^*\) has a closed form. Change the prior, the noise and the population and see where the optimum goes.

Population N (log scale)

Prior mean reward m

Prior spread τ (typical effect size)

Noise σ

Test size n per arm (log scale)

Try these

The plot shows the expected reward above the no-information value \(Nm\) (the second term of the formula). The error rate is \(\tfrac12-\tfrac1\pi\arctan(\tau\sqrt n/\sigma)\), the probability that the arm with the smaller true mean is deployed. The simulation draws \(\mu_0,\mu_1\) from the prior, the two sample means, deploys the arm with the larger sample mean, and averages the total reward over 20,000 experiments.

In the lecture notes