Interactive demo · Applied Statistical Learning in R and Python · Chapter 13

Benjamini–Hochberg on the Fund managers

Each of 2,000 fund managers has 50 months of excess returns, and a one-sample t-test asks whether the mean is zero. Controlling the family-wise error rate at any reasonable level rejects almost nothing. The Benjamini–Hochberg procedure controls instead the false discovery rate, the expected share of false positives among the rejections: sort the p-values, find the last one below the line q j/m, and reject it and every smaller one. Its threshold depends on all the p-values, so testing a different set of managers changes it.

FDR level q (and FWER level α = q)

Managers tested the first m

Try these

p-values: one-sample t-tests of H0j: μj = 0 with 49 degrees of freedom. BH rejects p(j) for j ≤ L = max{j: p(j) ≤ q j/m}; Bonferroni rejects p ≤ α/m; Holm stops at the first p(j) > α/(m + 1 − j). The q-value of a manager is the smallest q at which BH rejects it: the running minimum, from the right, of m p(k)/k.

In the lecture notes