For each of 2,308 genes, a two-sample t-statistic compares the expression of 29 patients in class 2 with 25 patients in class 4 of the Khan data. The textbook p-value refers it to a t distribution with 52 degrees of freedom, which assumes normal data. The permutation p-value assumes only that, under the null hypothesis, the class labels are exchangeable: shuffle the labels many times, recompute the statistic, and count how often it is at least as extreme as the observed one. For most genes the two agree; for a gene with one extreme value they do not. Pooling the shuffles over all genes estimates the false discovery rate without any p-value.
Statistic: the pooled two-sample t (13.11). Permutation p-value (13.12): the share of relabellings with |T*| ≥ |T|. The scatter in panel (c) shows, for every gene, the t-distribution p-value against a permutation p-value from 10,000 relabellings computed in advance (a value 0 is drawn at 1/10,000). The FDR curve uses relabellings computed in your browser, the same ones for every gene, and V̂ = (number of permuted |T*| ≥ c)/B (Algorithm 13.4).
code/ch13_multiple_testing.py (t_two_sample, perm_tstats, khan_permutations, plugin_fdr, Figures 13.7–13.9).