Interactive demo · Applied Statistical Learning in R and Python · Chapter 13

Permutation null versus the t distribution

For each of 2,308 genes, a two-sample t-statistic compares the expression of 29 patients in class 2 with 25 patients in class 4 of the Khan data. The textbook p-value refers it to a t distribution with 52 degrees of freedom, which assumes normal data. The permutation p-value assumes only that, under the null hypothesis, the class labels are exchangeable: shuffle the labels many times, recompute the statistic, and count how often it is at least as extreme as the observed one. For most genes the two agree; for a gene with one extreme value they do not. Pooling the shuffles over all genes estimates the false discovery rate without any p-value.

Gene 1 to 2,308

Relabel the 54 patients

Bottom panel

Try these

Statistic: the pooled two-sample t (13.11). Permutation p-value (13.12): the share of relabellings with |T*| ≥ |T|. The scatter in panel (c) shows, for every gene, the t-distribution p-value against a permutation p-value from 10,000 relabellings computed in advance (a value 0 is drawn at 1/10,000). The FDR curve uses relabellings computed in your browser, the same ones for every gene, and V̂ = (number of permuted |T*| ≥ c)/B (Algorithm 13.4).

In the lecture notes