Each of 2,000 fund managers has 50 months of excess returns, and a one-sample t-test asks whether the mean is zero. Controlling the family-wise error rate at any reasonable level rejects almost nothing. The Benjamini–Hochberg procedure controls instead the false discovery rate, the expected share of false positives among the rejections: sort the p-values, find the last one below the line q j/m, and reject it and every smaller one. Its threshold depends on all the p-values, so testing a different set of managers changes it.
p-values: one-sample t-tests of H0j: μj = 0 with 49 degrees of freedom. BH rejects p(j) for j ≤ L = max{j: p(j) ≤ q j/m}; Bonferroni rejects p ≤ α/m; Holm stops at the first p(j) > α/(m + 1 − j). The q-value of a manager is the smallest q at which BH rejects it: the running minimum, from the right, of m p(k)/k.
code/ch13_multiple_testing.py (bh, p_bh, holm, bonferroni, Figure 13.6).