Interactive demo · Applied Statistical Learning in R and Python · Chapter 13

The family-wise error rate: Bonferroni and Holm

Test m null hypotheses, each at level α. If m0 of them are true and the tests are independent, the chance of at least one false rejection, the family-wise error rate, is 1 − (1 − α)m0: almost 1 for a hundred tests. Bonferroni tests each hypothesis at α/m, which keeps the FWER below α whatever the dependence; Holm compares the sorted p-values with the rising thresholds α/(m + 1 − j), keeps the same guarantee and always rejects at least as many. Run experiments and watch both error rates and the power.

Number of tests m

Level α

Share of true nulls π0 = m0/m

Effect of the false nulls δ (standard errors)

Correlation between tests ρ

Run experiments

Try these

Each experiment draws m test statistics Zj = √ρ W + √(1 − ρ) ej + δj with W, ej independent N(0, 1); δj = 0 for the m0 true nulls and δ for the others; p-values are two-sided, 2Φ(−|Zj|). FWER = share of experiments with at least one true null rejected; power = average share of false nulls rejected. The curves in panel (a) are exact under independence.

In the lecture notes