Test m null hypotheses, each at level α. If m0 of them are true and the tests are independent, the chance of at least one false rejection, the family-wise error rate, is 1 − (1 − α)m0: almost 1 for a hundred tests. Bonferroni tests each hypothesis at α/m, which keeps the FWER below α whatever the dependence; Holm compares the sorted p-values with the rising thresholds α/(m + 1 − j), keeps the same guarantee and always rejects at least as many. Run experiments and watch both error rates and the power.
Each experiment draws m test statistics Zj = √ρ W + √(1 − ρ) ej + δj with W, ej independent N(0, 1); δj = 0 for the m0 true nulls and δ for the others; p-values are two-sided, 2Φ(−|Zj|). FWER = share of experiments with at least one true null rejected; power = average share of false nulls rejected. The curves in panel (a) are exact under independence.
code/ch13_multiple_testing.py (fwer_independent, bonferroni, holm, power_vs_fwer, Figures 13.2, 13.3 and 13.5).