If the two samples come from the same distribution, every way of splitting the pooled values into groups of sizes m and n is equally likely. The permutation distribution of a statistic T is its value on every split, and the p-value is the fraction of splits whose T is at least the observed one. With small samples the splits can all be listed, so nothing is approximated. Book data shows the whole permutation distribution for the small data sets of Section 10.5 and Exercise 10.7. Size under H0 simulates data for which the null hypothesis holds and counts how often each test rejects: the permutation test never exceeds α; the Wald test, which relies on large samples, can.
A split is a choice of which m of the N = m + n pooled values form the first sample; there are C(N, m) of them (the N! orderings of Section 10.5 give each split m!n! times, so the two descriptions agree). Statistics: |x̄ − ȳ|, the absolute difference of medians, and |W| with the Wald statistic of Example 10.8 (sample variances with divisor n − 1). The Monte Carlo p-value with B relabellings is (1 + #{Tb ≥ t})/(B + 1); relabelling b sorts N seeded uniforms and gives the first sample the m smallest. Size panel: data set r uses seeded normals z (draws rN + 1 to (r + 1)N), transformed to ez for the lognormal and multiplied by 3 in the second sample for "unequal spreads". The permutation test there uses |x̄ − ȳ| over all splits; the pooled t-test uses tN−2.
code/ch10_hypothesis_testing.py (perm_distribution, perm_pvalues, perm_mc, perm_size_sim); data: data/ch10-permutation.json.