Interactive demo · Statistical Methods for Modern Biology · Chapter 6

P-value hacking: optional stopping and the best of many statistics

Every study here is run under the null hypothesis: the observations are N(0, 1), so a 5% test should give a false positive in 5% of studies. Optional stopping: test after every new observation and stop as soon as p < 0.05. Choosing the statistic: compute several test statistics and report the smallest p-value. Both raise the false positive rate well above 5%, and for optional stopping it tends to 1 as the patience grows.

View

First look at n0

Look every d observations

Give up at nmax

Number of statistics K

Correlation ρ between them

Scoreboard

Try these

Exact optional-stopping probability: the density of the running sum, restricted to the paths that have not stopped yet, is carried on a grid and convolved with the increment density between looks (Armitage, McPherson and Rowe 1969). The best-of-K probability for equicorrelated normal statistics is a one-dimensional integral. Both are computed in your browser with the same algorithm as the R code.

In the lecture notes