"""Wasserman (2004), Chapter 10: Hypothesis Testing and p-values.

Companion code for Chapter 10 of L. Wasserman, All of Statistics: A Concise Course in Statistical
Inference (Springer, 2004), pp. 149-173, written from scratch with numpy, scipy and matplotlib. The
functions are short and meant to be read next to the lecture notes (../ch10-hypothesis-testing.html):

* Introduction, Example 10.2: the power function of the one-sided test of a Normal mean
  (power_normal) and its critical value.
* Section 10.1, the Wald test: the approximate power (10.6) (wald_power_approx), the exact power and
  size of the Wald test for a Normal mean, a Bernoulli probability and a Poisson mean, with the
  standard error estimated or computed under the null (Remark 10.5) (wald_exact_power), and a
  simulation of the test (wald_sim) that reproduces the browser demonstration number for number.
* Section 10.2, p-values: the Wald p-value (10.7), the exact distribution of the p-value under the
  null and under an alternative (pvalue_cdf_wald), and P(H0 | reject) for a test seen as a
  diagnostic test (posterior_null).
* Sections 10.3-10.4 and 10.6: Pearson's statistic and the likelihood ratio statistic for
  multinomial data (pearson_stat, lrt_multinomial; Mendel's peas, Examples 10.18 and 10.23), their
  simulated null distributions, and the exact size of the likelihood ratio tests of Exercises 10.13
  and 10.14 when the chi-squared limit of Theorem 10.22 is used with small samples (lrt_exact_size).
* Section 10.5, the permutation test: the exact permutation distribution over all splits
  (perm_distribution), p-values with the two conventions ">=" and ">" (perm_pvalues), the Monte
  Carlo version, and the size of the Wald, pooled t and permutation tests under the null in small
  samples (perm_size_sim, the second panel of the permutation demonstration).
* Section 10.7, multiple testing: Bonferroni and Benjamini-Hochberg (bonferroni, bh_threshold,
  exactly as written in the book) and a simulation of their family-wise error and false discovery
  rates (mt_sim).
* Section 10.8, goodness of fit: Pearson's statistic with the grouped (multinomial) MLE (10.9) and
  with the raw-data MLE, applied to the book's nerve data with an Exponential model (gof_*).
* Appendix 10.10.1, the Neyman-Pearson lemma: simulated likelihood ratio statistics and competitors
  for four pairs of simple hypotheses (np_sample, np_stats), the exact most powerful test where it
  has a closed form (np_exact), and empirical ROC curves (roc_points).
* Appendix 10.10.2, the t-test, and the exercises (ex_* functions).
* js_uniforms / js_normals: the seeded generator of the browser demonstrations (DK.rng in
  courses/demo-kit.js), so that the Monte Carlo numbers on the pages can be reproduced here.

Figures of the book reproduced (Wasserman 2004):
  10.1 the power function of Example 10.2;
  10.4 the two-sided Wald p-value as a Normal tail area (for the two statistics of Example 10.15);
  10.5 the p-value of a chi-squared test as an upper tail area (Mendel's peas, Example 10.18);
  10.6 the Benjamini-Hochberg procedure (Example 10.27, and the ten p-values of Example 10.28).
Figures skipped (schematic illustrations with nothing computed):
  10.2 two hand-drawn confidence intervals, one near and one far from theta_0 (statistical versus
       scientific significance); the idea is shown with real numbers in Figure 10.S1(a) instead;
  10.3 a step diagram of "reject or not" against alpha (the definition of the p-value); it is a
       drawing of Definition 10.11 with no data or model behind it.
Supplementary figures (numbered 10.S1 to 10.S7) illustrate the gaps filled in the notes:
  10.S1 the power of the Wald test: exact against (10.6), and the exact size in discrete models;
  10.S2 the distribution of p-values under the null and the alternative, and P(H0 | reject);
  10.S3 the chi-squared family, Pearson and likelihood ratio statistics for Mendel's peas, and the
        exact size of chi-squared-calibrated likelihood ratio tests;
  10.S4 the permutation test: exact permutation distributions and sizes in small samples;
  10.S5 multiple testing: family-wise error rate and false discovery rate by simulation;
  10.S6 the Neyman-Pearson lemma: ROC curves of most powerful tests and of competitors;
  10.S7 a goodness-of-fit test of the Exponential model for the nerve data (Theorem 10.29 and the
        Chernoff-Lehmann bound).
Run `python3 ch10_hypothesis_testing.py --list`, `python3 ch10_hypothesis_testing.py` (all figures,
about ten seconds on one CPU thread) or `python3 ch10_hypothesis_testing.py --fig 10.S4`.
Files go to ../img/ch10/; the data of the permutation demonstration go to ../data/ch10-permutation.json.
"""
import itertools
import json
import math
import os

import numpy as np
from scipy import optimize, special, stats

import common
from common import BLUE, GREEN, GREY, MAGENTA, ORANGE, PURPLE, RED, savefig

CH = 10
M32 = 0xFFFFFFFF
HERE = os.path.dirname(os.path.abspath(__file__))


# =========================================================================== the demos' generator
def _imul(a, b):
    """32-bit integer multiplication with wrap-around, as Math.imul (result as unsigned)."""
    return (a * b) & M32


class JSRandom:
    """The seeded random numbers of the browser demos (DK.rng in courses/demo-kit.js), one call at a
    time: u() is mulberry32 and norm() the Box-Muller transform that returns r cos(theta) and keeps
    r sin(theta) for the next call, exactly as in JavaScript."""

    def __init__(self, seed):
        self.a = int(seed) & M32
        self.spare = None

    def u(self):
        a = (self.a + 0x6D2B79F5) & M32
        self.a = a
        t = _imul(a ^ (a >> 15), 1 | a)
        t = ((t + _imul(t ^ (t >> 7), 61 | t)) & M32) ^ t
        return ((t ^ (t >> 14)) & M32) / 4294967296.0

    def norm(self):
        if self.spare is not None:
            s, self.spare = self.spare, None
            return s
        x = 0.0
        while x == 0.0:
            x = self.u()
        r = math.sqrt(-2.0 * math.log(x))
        th = 2.0 * math.pi * self.u()
        self.spare = r * math.sin(th)
        return r * math.cos(th)


def js_uniforms(seed, n, start=0):
    """Uniforms number start+1, ..., start+n of DK.rng(seed), vectorised: mulberry32 adds a constant
    to its state at each call, so the j-th output is a hash of seed + j * 0x6D2B79F5 (mod 2^32)."""
    j = np.arange(start + 1, start + n + 1, dtype=np.uint64)
    m = np.uint64(M32)
    a = (np.uint64(int(seed) & M32) + j * np.uint64(0x6D2B79F5)) & m
    t = ((a ^ (a >> np.uint64(15))) * (np.uint64(1) | a)) & m
    t = ((t + (((t ^ (t >> np.uint64(7))) * (np.uint64(61) | t)) & m)) & m) ^ t
    return ((t ^ (t >> np.uint64(14))) & m).astype(np.float64) / 4294967296.0


def js_normals(seed, n, start=0):
    """n consecutive DK.rng(seed).norm() values drawn after `start` uniforms with no normal cached
    (Box-Muller pairs: cos first, then the cached sin)."""
    npair = (n + 1) // 2
    u = js_uniforms(seed, 2 * npair, start)
    u1, u2 = u[0::2], u[1::2]
    if np.any(u1 == 0.0):
        raise RuntimeError("a zero uniform: use JSRandom for this stream")
    r = np.sqrt(-2.0 * np.log(u1))
    th = 2.0 * np.pi * u2
    out = np.empty(2 * npair)
    out[0::2] = r * np.cos(th)
    out[1::2] = r * np.sin(th)
    return out[:n]


def seq_sum(X):
    """Row sums of a 2-D array added left to right, the order a JavaScript loop uses (numpy's own
    sum is pairwise and can differ in the last bit)."""
    s = np.zeros(X.shape[0])
    for j in range(X.shape[1]):
        s = s + X[:, j]
    return s


def zq(alpha):
    """Upper alpha quantile of N(0, 1): z_alpha = Phi^{-1}(1 - alpha)."""
    return stats.norm.isf(alpha)


# =========================================================================== Example 10.2
def power_normal(mu, n, sigma=1.0, alpha=0.05, mu0=0.0):
    """Power function of Example 10.2: reject H0: mu <= mu0 when sqrt(n)(xbar - mu0)/sigma > z_alpha,
    beta(mu) = 1 - Phi(z_alpha - sqrt(n)(mu - mu0)/sigma); the size is beta(mu0) = alpha."""
    return stats.norm.sf(zq(alpha) - np.sqrt(n) * (np.asarray(mu, float) - mu0) / sigma)


# =========================================================================== Section 10.1: the Wald test
def wald_power_approx(theta, theta0, se, alpha=0.05, side="two"):
    """The approximate power (10.6) of the size-alpha Wald test when the true value is theta and the
    standard error of the estimator is se. side='upper' gives the one-sided version."""
    d = (theta0 - np.asarray(theta, float)) / se
    if side == "upper":
        return stats.norm.sf(zq(alpha) + d)
    z = zq(alpha / 2)
    return stats.norm.sf(d + z) + stats.norm.cdf(d - z)


def true_se(model, theta, n):
    """Standard error of the MLE at the true value: 1/sqrt(n) (Normal, sigma = 1), sqrt(p(1-p)/n),
    sqrt(lambda/n)."""
    theta = np.asarray(theta, float)
    if model == "normal":
        return 1.0 / np.sqrt(n) + 0 * theta
    if model == "bernoulli":
        return np.sqrt(theta * (1 - theta) / n)
    return np.sqrt(theta / n)


def wald_reject_from_S(model, S, n, theta0, alpha, se_kind="hat", side="two"):
    """Reject indicator of the Wald test as a function of the sufficient statistic S (the sum of the
    observations; for the Normal model S is the sum of n N(theta, 1) values). se_kind = 'hat' uses the
    estimated standard error, 'null' the one computed at theta0 (Remark 10.5). A zero estimated
    standard error means the estimate has no spread: reject if and only if the estimate differs
    from theta0."""
    S = np.asarray(S, float)
    est = S / n
    if model == "normal":
        se = np.full_like(est, 1.0 / math.sqrt(n))
    elif model == "bernoulli":
        se = np.sqrt(est * (1 - est) / n) if se_kind == "hat" else np.full_like(est, math.sqrt(theta0 * (1 - theta0) / n))
    else:
        se = np.sqrt(est / n) if se_kind == "hat" else np.full_like(est, math.sqrt(theta0 / n))
    crit = zq(alpha) if side == "upper" else zq(alpha / 2)
    with np.errstate(divide="ignore", invalid="ignore"):
        W = (est - theta0) / se
    W = np.where(se > 0, W, np.where(est > theta0, np.inf, np.where(est < theta0, -np.inf, 0.0)))
    return (W > crit) if side == "upper" else (np.abs(W) > crit)


def wald_exact_power(model, theta, theta0, n, alpha=0.05, se_kind="hat", side="two"):
    """Exact rejection probability of the Wald test at the true value theta (the size when
    theta = theta0): a closed form for the Normal mean, a finite sum over the Binomial or Poisson
    distribution of S otherwise."""
    if model == "normal":
        d = math.sqrt(n) * (theta - theta0)
        if side == "upper":
            return float(stats.norm.sf(zq(alpha) - d))
        z = zq(alpha / 2)
        return float(stats.norm.sf(z - d) + stats.norm.cdf(-z - d))
    if model == "bernoulli":
        s = np.arange(n + 1)
        w = stats.binom.pmf(s, n, theta)
    else:
        lam = n * theta
        smax = int(lam + 12 * math.sqrt(lam) + 30)
        s = np.arange(smax + 1)
        w = stats.poisson.pmf(s, lam)
    return float(np.sum(w * wald_reject_from_S(model, s, n, theta0, alpha, se_kind, side)))


def poisson_inverse(u, lam):
    """Poisson(lam) draws by inversion, with the cumulative sums formed exactly as the demo does
    (p = exp(-lam), then p *= lam / k, F += p), so that the draws agree with JavaScript bit for bit."""
    p = math.exp(-lam)
    F = [p]
    k = 0
    while F[-1] < 1 - 1e-15 and k < 1000:
        k += 1
        p = p * lam / k
        F.append(F[-1] + p)
    F = np.array(F)
    idx = np.searchsorted(F, u, side="right")
    return np.minimum(idx, len(F) - 1)


def wald_sim(model, theta, theta0, n, alpha=0.05, R=2000, seed=1, se_kind="hat", side="two"):
    """Simulate R data sets of size n at the true value theta with DK.rng(seed) (data set r uses
    draws r n + 1, ..., (r + 1) n: normals for the Normal model, uniforms otherwise) and return the
    fraction of Wald rejections and the vector of sums S."""
    if model == "normal":
        X = theta + js_normals(seed, R * n).reshape(R, n)
    else:
        U = js_uniforms(seed, R * n).reshape(R, n)
        X = (U < theta).astype(float) if model == "bernoulli" else poisson_inverse(U, theta).astype(float)
    S = seq_sum(X)
    rej = wald_reject_from_S(model, S, n, theta0, alpha, se_kind, side)
    return float(rej.mean()), S


# =========================================================================== Section 10.2: p-values
def wald_pvalue(w):
    """(10.7): the two-sided Wald p-value 2 Phi(-|w|)."""
    return 2 * stats.norm.cdf(-np.abs(w))


def pvalue_cdf_wald(u, delta):
    """P(p-value <= u) for the two-sided Wald test of a Normal mean when the standardised distance
    from the null is delta = sqrt(n)(theta - theta0)/sigma: the power at level u. delta = 0 gives u."""
    z = stats.norm.isf(np.asarray(u, float) / 2)
    return stats.norm.sf(z - delta) + stats.norm.cdf(-z - delta)


def pvalue_pdf_wald(u, delta):
    """Density of the two-sided Wald p-value under the alternative delta (derivative of pvalue_cdf_wald)."""
    z = stats.norm.isf(np.asarray(u, float) / 2)
    return (stats.norm.pdf(z - delta) + stats.norm.pdf(z + delta)) / (2 * stats.norm.pdf(z))


def posterior_null(pi0, alpha, power):
    """P(H0 | reject) when a fraction pi0 of the hypotheses tested are true nulls, the test has size
    alpha and power `power` (Bayes' theorem, as for a diagnostic test)."""
    return pi0 * alpha / (pi0 * alpha + (1 - pi0) * power)


def bayes_factor_bound(z):
    """Lower bound on P(data | H0) / P(data | H1) over all priors on the alternative for a Normal
    statistic z: exp(-z^2/2)."""
    return np.exp(-0.5 * np.asarray(z, float) ** 2)


# =========================================================================== Sections 10.3, 10.4, 10.6
MENDEL_X = np.array([315, 101, 108, 32])
MENDEL_P0 = np.array([9, 3, 3, 1]) / 16.0


def pearson_stat(X, p0):
    """Pearson's statistic (Definition 10.16): sum (X_j - n p0j)^2 / (n p0j); works row-wise."""
    X = np.asarray(X, float)
    n = X.sum(axis=-1, keepdims=True)
    E = n * p0
    return ((X - E) ** 2 / E).sum(axis=-1)


def lrt_multinomial(X, p0):
    """The likelihood ratio statistic of Example 10.23: 2 sum X_j log(X_j / (n p0j)), with 0 log 0 = 0."""
    X = np.asarray(X, float)
    n = X.sum(axis=-1, keepdims=True)
    with np.errstate(divide="ignore", invalid="ignore"):
        t = np.where(X > 0, X * np.log(X / (n * p0)), 0.0)
    return 2 * t.sum(axis=-1)


def mendel_null_sim(R=20000, seed=1018):
    """Pearson and likelihood ratio statistics for R multinomial(556, p0) samples."""
    rng = np.random.default_rng(seed)
    X = rng.multinomial(MENDEL_X.sum(), MENDEL_P0, size=R)
    return pearson_stat(X, MENDEL_P0), lrt_multinomial(X, MENDEL_P0)


def lrt_normal_mean(T, n):
    """Exercise 10.13: with sigma unknown the LRT statistic for mu = mu0 is n log(1 + T^2/(n - 1)),
    T the t statistic."""
    return n * np.log1p(np.asarray(T, float) ** 2 / (n - 1))


def lrt_normal_sd(s, n):
    """Exercise 10.14: with mu unknown and s = sigma_hat^2 / sigma0^2 (sigma_hat the MLE), the LRT
    statistic for sigma = sigma0 is n (s - 1 - log s)."""
    s = np.asarray(s, float)
    return n * (s - 1 - np.log(s))


def lrt_exact_size(kind, n, alpha=0.05):
    """Exact size of the LRT of Exercise 10.13 ('mean') or 10.14 ('sd') for Normal data when H0 is
    rejected for lambda > chi^2_{1, alpha}, as Theorem 10.22 suggests."""
    c = stats.chi2.isf(alpha, 1)
    if kind == "mean":
        # n log(1 + T^2/(n-1)) > c  <=>  |T| > sqrt((n-1)(exp(c/n) - 1)),  T ~ t_{n-1}
        t = math.sqrt((n - 1) * math.expm1(c / n))
        return float(2 * stats.t.sf(t, n - 1))
    # n (s - 1 - log s) > c with n s ~ chi^2_{n-1}: s outside the two roots of s - 1 - log s = c/n
    g = lambda s: s - 1 - math.log(s) - c / n
    lo = optimize.brentq(g, 1e-12, 1.0)
    hi = optimize.brentq(g, 1.0, 1e3)
    return float(stats.chi2.cdf(n * lo, n - 1) + stats.chi2.sf(n * hi, n - 1))


def wald_sd_exact_size(n, alpha=0.05):
    """Exact size of the Wald test of sigma = sigma0 for Normal data with mu unknown (Exercise 10.14):
    W = (sigma_hat - sigma0) / (sigma_hat / sqrt(2n)), sigma_hat the MLE, n sigma_hat^2/sigma0^2 ~ chi^2_{n-1}."""
    c = zq(alpha / 2) / math.sqrt(2 * n)
    # |1 - sigma0/sigma_hat| > c  <=>  r = sigma_hat/sigma0 < 1/(1+c)  or  (c < 1 and r > 1/(1-c))
    lo = (1 / (1 + c)) ** 2
    p = stats.chi2.cdf(n * lo, n - 1)
    if c < 1:
        p += stats.chi2.sf(n * (1 / (1 - c)) ** 2, n - 1)
    return float(p)


def binom_lrt_stat(S, n, p0):
    """Exercise 10.15: lambda = 2[S log(phat/p0) + (n-S) log((1-phat)/(1-p0))], with 0 log 0 = 0."""
    S = np.asarray(S, float)
    ph = S / n
    with np.errstate(divide="ignore", invalid="ignore"):
        a = np.where(S > 0, S * np.log(ph / p0), 0.0)
        b = np.where(S < n, (n - S) * np.log((1 - ph) / (1 - p0)), 0.0)
    return 2 * (a + b)


def binom_exact_size(kind, n, p0, alpha=0.05):
    """Exact size of the Wald ('wald', estimated se) or likelihood ratio ('lrt', chi^2_1 cut-off) test
    of p = p0 for X ~ Binomial(n, p0)."""
    s = np.arange(n + 1)
    w = stats.binom.pmf(s, n, p0)
    if kind == "wald":
        rej = wald_reject_from_S("bernoulli", s, n, p0, alpha)
    else:
        rej = binom_lrt_stat(s, n, p0) > stats.chi2.isf(alpha, 1)
    return float(np.sum(w * rej))


def cl_eigen(rate, edges):
    """Chernoff-Lehmann: with the raw-data MLE, Q tends to chi^2_{k-2} + (1 - I_g/I) chi^2_1 for a
    one-parameter model, where I_g = sum p_j'(rate)^2 / p_j is the Fisher information of the grouped
    counts and I = 1/rate^2 that of one Exponential observation. Returns 1 - I_g/I."""
    e = edges[np.isfinite(edges)]
    F = 1 - np.exp(-rate * e)
    dF = e * np.exp(-rate * e)                     # d/d rate of F at the finite edges
    p = np.diff(np.r_[F, 1.0])
    dp = np.diff(np.r_[dF, 0.0])
    Ig = np.sum(dp ** 2 / p)
    return float(1 - Ig * rate ** 2)


# =========================================================================== Section 10.5: permutation test
TOY = (np.array([1.0, 9.0]), np.array([3.0]))
GENE1 = (np.array([230, -1350, -1580, -400, -760.0]), np.array([970, 110, -50, -190, -200.0]))
GENE2 = (np.array([470, -850, -0.8, -280, 120.0]), np.array([390, -1730, -1360, -1, -330.0]))
TWAIN = np.array([.225, .262, .217, .240, .230, .229, .235, .217])
SNODGRASS = np.array([.209, .205, .196, .210, .202, .207, .224, .223, .220, .201])


def two_sample_stat(x, y, stat):
    """|difference of means|, |difference of medians| or |Welch-type Wald statistic| (Example 10.8,
    with the n - 1 sample variances)."""
    if stat == "mean":
        return abs(np.mean(x) - np.mean(y))
    if stat == "median":
        return abs(np.median(x) - np.median(y))
    se = math.sqrt(np.var(x, ddof=1) / len(x) + np.var(y, ddof=1) / len(y))
    return abs(np.mean(x) - np.mean(y)) / se


def perm_distribution(x, y, stat="mean"):
    """The exact permutation distribution: the statistic for every way of choosing which m of the
    N = m + n pooled values form the first sample (C(N, m) equally likely splits; the N! orderings
    of Section 10.5 give each split m! n! times). The observed split comes first."""
    z = np.concatenate([x, y])
    N, m = len(z), len(x)
    out = []
    for c in itertools.combinations(range(N), m):
        mask = np.zeros(N, bool)
        mask[list(c)] = True
        out.append(two_sample_stat(z[mask], z[~mask], stat))
    return np.array(out)


def perm_pvalues(T, tobs, rel=1e-9):
    """p-values from a permutation distribution T: P0(T >= tobs) (valid) and P0(T > tobs) (the
    convention printed in Section 10.5). Values within rel * |tobs| count as ties."""
    tol = rel * max(1.0, abs(tobs))
    return float(np.mean(T >= tobs - tol)), float(np.mean(T > tobs + tol))


def perm_mc(x, y, stat, B, seed):
    """Monte Carlo permutation p-value with B random relabellings: relabelling b sorts N uniforms of
    DK.rng(seed) (draws b N + 1, ..., (b + 1) N) and gives the first sample the m smallest keys.
    Returns (1 + #{T_b >= tobs}) / (B + 1) and the book's #{T_b > tobs} / B."""
    z = np.concatenate([x, y])
    N, m = len(z), len(x)
    tobs = two_sample_stat(x, y, stat)
    U = js_uniforms(seed, B * N).reshape(B, N)
    order = np.argsort(U, axis=1, kind="stable")
    T = np.array([two_sample_stat(z[o[:m]], z[o[m:]], stat) for o in order])
    tol = 1e-9 * max(1.0, abs(tobs))
    return (1 + np.sum(T >= tobs - tol)) / (B + 1), float(np.mean(T > tobs + tol))


def wald_two_sample(x, y, ddof=1):
    """Wald test of equal means (Example 10.8): W = (xbar - ybar)/se with se^2 = s1^2/m + s2^2/n
    (ddof = 1: the sample variances; ddof = 0: the plug-in variances of Exercise 10.7)."""
    se = math.sqrt(np.var(x, ddof=ddof) / len(x) + np.var(y, ddof=ddof) / len(y))
    W = (np.mean(x) - np.mean(y)) / se
    return W, float(wald_pvalue(W)), se


def _splits(N, m):
    """All C(N, m) splits as a boolean matrix (row k: True for the indices of the first sample)."""
    combos = list(itertools.combinations(range(N), m))
    M = np.zeros((len(combos), N), bool)
    for k, c in enumerate(combos):
        M[k, list(c)] = True
    return M


SIZE_DISTS = ("normal", "lognormal", "unequal")


def size_data(dist, m, n, R, seed):
    """R null data sets for the size panel of the permutation demo, from DK.rng(seed): data set r uses
    normals r N + 1, ..., (r + 1) N (N = m + n), the first m for sample 1. 'normal': N(0, 1) values z;
    'lognormal': exp(z), a skewed distribution; 'unequal': z for sample 1 and 3 z for sample 2 (equal
    means but unequal spreads, so the null of the permutation test, F_X = F_Y, is false)."""
    N = m + n
    Z = js_normals(seed, R * N).reshape(R, N)
    if dist == "normal":
        return Z
    if dist == "lognormal":
        return np.exp(Z)
    D = Z.copy()
    D[:, m:] *= 3.0
    return D


def perm_size_sim(dist, m, n, alpha=0.05, R=2000, seed=7):
    """Rejection rates under H0 of four two-sided tests of equal means at level alpha: the Wald test
    (z with the n - 1 sample variances), the pooled t-test, and the exact permutation test of
    |difference of means| with the valid p-value (>=) and with the book's (>). Each permutation
    p-value is computed by full enumeration of the C(N, m) splits. Returns a dict of rates."""
    D = size_data(dist, m, n, R, seed)
    N = m + n
    M = _splits(N, m).astype(float)                     # K x N
    tot = seq_sum(D)
    s1 = D @ M.T                                         # R x K sums of the first sample
    diff = np.abs(s1 / m - (tot[:, None] - s1) / n)
    tobs = diff[:, 0]
    tol = 1e-9 * np.maximum(1.0, tobs)
    p_ge = np.mean(diff >= (tobs - tol)[:, None], axis=1)
    p_gt = np.mean(diff > (tobs + tol)[:, None], axis=1)
    x, y = D[:, :m], D[:, m:]
    v1, v2 = x.var(axis=1, ddof=1), y.var(axis=1, ddof=1)
    W = (x.mean(1) - y.mean(1)) / np.sqrt(v1 / m + v2 / n)
    sp = ((m - 1) * v1 + (n - 1) * v2) / (N - 2)
    T = (x.mean(1) - y.mean(1)) / np.sqrt(sp * (1 / m + 1 / n))
    return {"wald": float(np.mean(np.abs(W) > zq(alpha / 2))),
            "t": float(np.mean(np.abs(T) > stats.t.isf(alpha / 2, N - 2))),
            "perm": float(np.mean(p_ge <= alpha)),
            "perm_gt": float(np.mean(p_gt <= alpha))}


# =========================================================================== Section 10.7: multiple testing
def bonferroni(p, alpha=0.05):
    """The Bonferroni method: reject H0i when P_i < alpha / m."""
    p = np.asarray(p, float)
    return p < alpha / p.shape[-1]


def bh_threshold(p, alpha=0.05, dependent=False):
    """The Benjamini-Hochberg threshold T = P_(R), R = max{i: P_(i) < i alpha / (C_m m)}, exactly as
    in the book (C_m = 1 for independent p-values, sum_{i<=m} 1/i otherwise). Returns T (or 0 if no
    i qualifies, in which case nothing is rejected)."""
    p = np.sort(np.asarray(p, float))
    m = len(p)
    cm = np.sum(1.0 / np.arange(1, m + 1)) if dependent else 1.0
    ok = np.nonzero(p < np.arange(1, m + 1) * alpha / (cm * m))[0]
    return float(p[ok[-1]]) if len(ok) else 0.0


def bh_reject(p, alpha=0.05, dependent=False):
    """Reject all H0i with P_i <= T (T the BH threshold); nothing if no p-value is under the line."""
    T = bh_threshold(p, alpha, dependent)
    p = np.asarray(p, float)
    return (p <= T) if T > 0 else np.zeros(p.shape, bool)


def mt_sim(m, m0, mu1=3.0, rho=0.0, alpha=0.05, R=4000, seed=107):
    """Simulate R families of m one-sided z-tests, the first m0 nulls true (mean 0) and the rest with
    mean mu1; the statistics are equicorrelated with correlation rho >= 0. Returns the FWER and the
    FDR of uncorrected testing, Bonferroni, BH and BH with C_m (Benjamini-Yekutieli)."""
    rng = np.random.default_rng(seed)
    Z = math.sqrt(1 - rho) * rng.standard_normal((R, m)) + math.sqrt(rho) * rng.standard_normal((R, 1))
    Z[:, m0:] += mu1
    P = stats.norm.sf(Z)
    out = {}
    rules = {"none": lambda p: p < alpha, "bonferroni": lambda p: bonferroni(p, alpha),
             "bh": lambda p: bh_reject(p, alpha), "by": lambda p: bh_reject(p, alpha, True)}
    for k, rule in rules.items():
        V = np.zeros(R)
        Rr = np.zeros(R)
        for r in range(R):
            rej = rule(P[r])
            V[r] = rej[:m0].sum()
            Rr[r] = rej.sum()
        fdp = np.where(Rr > 0, V / np.maximum(Rr, 1), 0.0)
        out[k] = {"fwer": float(np.mean(V > 0)), "fdr": float(np.mean(fdp)), "power": float(np.mean(Rr - V) / max(m - m0, 1))}
    return out


EX_10_28 = np.array([0.00017, 0.00448, 0.00671, 0.00907, 0.01220, 0.33626, 0.39341, 0.53882, 0.58125, 0.98617])
FIG_10_6_P = np.array([0.010, 0.015, 0.035, 0.042, 0.060, 0.090])


# =========================================================================== Section 10.8: goodness of fit
def nerve():
    """The nerve data of the book (nerve.dat): 799 waiting times (seconds) between nerve pulses."""
    with open(common.aos_file("nerve.dat")) as f:
        return np.array(f.read().split(), float)


NERVE_EDGES = np.array([0.0, 0.025, 0.055, 0.085, 0.125, 0.175, 0.235, 0.315, 0.435, 0.655, np.inf])


def exp_cell_probs(rate, edges):
    """p_j(rate) = P(X in [e_j, e_{j+1})) for an Exponential(rate) model."""
    F = 1 - np.exp(-rate * edges)
    return np.diff(F)


def gof_grouped_mle(counts, edges):
    """The multinomial (grouped-data) MLE of the Exponential rate: maximise sum N_j log p_j(rate),
    the likelihood Q(theta) of Section 10.8."""
    counts = np.asarray(counts, float)
    f = lambda lr: -np.sum(counts * np.log(np.maximum(exp_cell_probs(math.exp(lr), edges), 1e-300)))
    r = optimize.minimize_scalar(f, bounds=(-5, 8), method="bounded", options={"xatol": 1e-10})
    return math.exp(r.x)


def gof_stat(counts, rate, edges):
    """Pearson's statistic (10.9) with the cell probabilities at the given rate."""
    counts = np.asarray(counts, float)
    E = counts.sum() * exp_cell_probs(rate, edges)
    return float(np.sum((counts - E) ** 2 / E))


def gof_nerve():
    """Theorem 10.29 for the nerve data: counts, grouped and raw MLEs, both statistics and p-values
    against chi^2_{k-1-s} and chi^2_{k-1} (k cells, s = 1 parameter)."""
    x = nerve()
    counts = np.histogram(x, NERVE_EDGES)[0]
    k = len(counts)
    rg = gof_grouped_mle(counts, NERVE_EDGES)
    rr = 1 / x.mean()
    Qg, Qr = gof_stat(counts, rg, NERVE_EDGES), gof_stat(counts, rr, NERVE_EDGES)
    return dict(x=x, counts=counts, k=k, rate_grouped=rg, rate_raw=rr, Q_grouped=Qg, Q_raw=Qr,
                p_grouped=float(stats.chi2.sf(Qg, k - 2)), p_raw_low=float(stats.chi2.sf(Qr, k - 2)),
                p_raw_high=float(stats.chi2.sf(Qr, k - 1)))


def gof_null_sim(n, rate, R=2000, seed=1029):
    """Null distribution of Q with the grouped MLE and with the raw MLE for Exponential samples of
    size n (the data are continuous here, so the raw MLE is 1 / mean)."""
    rng = np.random.default_rng(seed)
    Qg, Qr = np.empty(R), np.empty(R)
    for r in range(R):
        x = rng.exponential(1 / rate, n)
        c = np.histogram(x, NERVE_EDGES)[0]
        Qg[r] = gof_stat(c, gof_grouped_mle(c, NERVE_EDGES), NERVE_EDGES)
        Qr[r] = gof_stat(c, 1 / x.mean(), NERVE_EDGES)
    return Qg, Qr


# =========================================================================== Appendix: Neyman-Pearson
NP_FAMILIES = ("normal", "laplace", "cauchy", "scale")
TINY = 2.0 ** -33


def np_sample(family, hyp, n, R, seed, delta):
    """R samples of size n under H0 (hyp=0) or H1 (hyp=1) from DK.rng(seed); the H0 samples use the
    first R n draws and the H1 samples the next R n (normals for 'normal' and 'scale', uniforms
    shifted by 2^-33 otherwise). H0: location 0 (or scale 1); H1: location delta (or scale delta)."""
    start = hyp * R * n
    if family in ("normal", "scale"):
        # normals come in Box-Muller pairs: draw the whole H0+H1 stream and cut
        Z = js_normals(seed, 2 * R * n)[start:start + R * n].reshape(R, n)
        if family == "normal":
            return Z + (delta if hyp else 0.0)
        return Z * (delta if hyp else 1.0)
    U = js_uniforms(seed, R * n, start).reshape(R, n) + TINY
    if family == "laplace":
        X = np.where(U < 0.5, np.log(2 * U), -np.log(2 - 2 * U))
    else:
        X = np.tan(np.pi * (U - 0.5))
    return X + (delta if hyp else 0.0)


def np_stats(family, X, delta):
    """(log likelihood ratio log L(theta1)/L(theta0), competitor statistic) for each row of X, both
    large under H1. Competitors: the sample median (normal, cauchy), the sample mean (laplace) and
    the mean absolute value (scale)."""
    if family == "normal":
        llr = seq_sum(delta * X - 0.5 * delta * delta + 0 * X)
        comp = np.median(X, axis=1)
    elif family == "laplace":
        llr = seq_sum(np.abs(X) - np.abs(X - delta))
        comp = seq_sum(X) / X.shape[1]
    elif family == "cauchy":
        llr = seq_sum(np.log((1 + X * X) / (1 + (X - delta) * (X - delta))))
        comp = np.median(X, axis=1)
    else:
        llr = seq_sum(0.5 * X * X * (1 - 1 / (delta * delta)) - np.log(delta) + 0 * X)
        comp = seq_sum(np.abs(X)) / X.shape[1]
    return llr, comp


def np_exact(family, n, delta, alpha):
    """Exact power of the most powerful size-alpha test where it has a closed form: the Normal mean
    (reject for large xbar) and the Normal scale (reject for large sum x^2, chi^2_n under H0)."""
    if family == "normal":
        return float(stats.norm.sf(zq(alpha) - math.sqrt(n) * delta))
    if family == "scale":
        c = stats.chi2.isf(alpha, n)
        return float(stats.chi2.sf(c / delta ** 2, n))
    return None


def mc_power(s0, s1, alpha):
    """Power at size alpha estimated from simulated statistics: the critical value is the order
    statistic of the H0 sample with exactly floor(alpha R) H0 values above it."""
    R = len(s0)
    k = int(math.floor(alpha * R + 1e-9))
    c = np.sort(s0)[R - k - 1]
    return float(np.mean(s1 > c)), c


def roc_points(s0, s1, npts=201):
    """Empirical ROC curve: (size, power) as the critical value runs over the H0 order statistics."""
    R = len(s0)
    a = np.sort(s0)
    s1s = np.sort(s1)
    ks = np.unique(np.round(np.linspace(0, R - 1, npts)).astype(int))
    size = ks / R
    power = 1 - np.searchsorted(s1s, a[R - ks - 1], side="right") / len(s1)
    return np.r_[size, 1.0], np.r_[power, 1.0]


def np_mc(family, n, delta, alpha=0.05, R=5000, seed=3):
    """Simulated power at size alpha of the likelihood ratio test and of the competitor (the numbers of
    the Neyman-Pearson demo)."""
    X0 = np_sample(family, 0, n, R, seed, delta)
    X1 = np_sample(family, 1, n, R, seed, delta)
    l0, c0 = np_stats(family, X0, delta)
    l1, c1 = np_stats(family, X1, delta)
    pl, kl = mc_power(l0, l1, alpha)
    pc, kc = mc_power(c0, c1, alpha)
    return dict(llr0=l0, llr1=l1, comp0=c0, comp1=c1, power_lr=pl, power_comp=pc, k_lr=kl, k_comp=kc,
                exact=np_exact(family, n, delta, alpha))


# =========================================================================== exercises
def ex5(n=20, alpha=0.05):
    """Exercise 10.5: Uniform(0, theta), reject theta = 1/2 when max > c."""
    c = 0.5 * (1 - alpha) ** (1 / n)
    return dict(c=c, p048=1 - (0.48 / 0.5) ** n, p052=0.0)


def ex6():
    """Exercise 10.6: 922 of 1919 deaths in the week before Passover."""
    n, x = 1919, 922
    th = x / n
    se = math.sqrt(th * (1 - th) / n)
    W = (th - 0.5) / se
    return dict(theta=th, se=se, W=W, p=float(wald_pvalue(W)), ci=(th - 1.96 * se, th + 1.96 * se),
                p_exact=float(stats.binomtest(x, n, 0.5).pvalue))


def ex7(B=10000, seed=7):
    """Exercise 10.7: Twain against Snodgrass."""
    W0, p0, se0 = wald_two_sample(TWAIN, SNODGRASS, ddof=0)
    W1, p1, se1 = wald_two_sample(TWAIN, SNODGRASS, ddof=1)
    d = TWAIN.mean() - SNODGRASS.mean()
    T = perm_distribution(TWAIN, SNODGRASS, "mean")
    pge, pgt = perm_pvalues(T, T[0])
    pmc = perm_mc(TWAIN, SNODGRASS, "mean", B, seed)
    return dict(d=d, W=W0, p=p0, se=se0, ci=(d - 1.96 * se0, d + 1.96 * se0), W1=W1, p1=p1, nsplit=len(T),
                p_perm=pge, p_perm_gt=pgt, count=int(round(pge * len(T))), p_mc=pmc)


EX10_TABLE = np.array([[55, 33, 70, 49], [141, 145, 139, 161]])   # rows Chinese, Jewish; weeks -2, -1, 1, 2


def ex10():
    """Exercise 10.10: chi-squared and likelihood ratio tests of homogeneity of the 2 x 4 table, and the
    before/after split for each group."""
    O = EX10_TABLE.astype(float)
    E = O.sum(1, keepdims=True) * O.sum(0, keepdims=True) / O.sum()
    X2 = float(((O - E) ** 2 / E).sum())
    G2 = float(2 * (O * np.log(O / E)).sum())
    before_after = O[:, :2].sum(1) / O.sum(1)
    # Chinese: before vs after (binomial 1/2)
    out = dict(X2=X2, p=float(stats.chi2.sf(X2, 3)), G2=G2, pG=float(stats.chi2.sf(G2, 3)), frac_before=before_after)
    # 2 x 2 collapse: before/after by group
    O2 = np.c_[O[:, :2].sum(1), O[:, 2:].sum(1)]
    E2 = O2.sum(1, keepdims=True) * O2.sum(0, keepdims=True) / O2.sum()
    X22 = float(((O2 - E2) ** 2 / E2).sum())
    out.update(X2_2x2=X22, p_2x2=float(stats.chi2.sf(X22, 1)))
    for g, row in zip(("chinese", "jewish"), O2):
        th = row[0] / row.sum()
        se = math.sqrt(th * (1 - th) / row.sum())
        out[g] = dict(theta=th, W=(th - 0.5) / se, p=float(wald_pvalue((th - 0.5) / se)))
    return out


EX11 = {"placebo": (80, 45), "chlorpromazine": (75, 26), "dimenhydrinate": (85, 52),
        "pentobarbital100": (67, 35), "pentobarbital150": (85, 37)}


def ex11(alpha=0.05):
    """Exercise 10.11: each drug against placebo (Wald test for p_drug - p_placebo), odds ratios, and
    the Bonferroni and BH adjustments over the four comparisons."""
    n0, x0 = EX11["placebo"]
    p0 = x0 / n0
    rows = {}
    for k in ("chlorpromazine", "dimenhydrinate", "pentobarbital100", "pentobarbital150"):
        n1, x1 = EX11[k]
        p1 = x1 / n1
        se = math.sqrt(p1 * (1 - p1) / n1 + p0 * (1 - p0) / n0)
        W = (p1 - p0) / se
        OR = (p1 / (1 - p1)) / (p0 / (1 - p0))
        se_log = math.sqrt(1 / x1 + 1 / (n1 - x1) + 1 / x0 + 1 / (n0 - x0))
        rows[k] = dict(p1=p1, W=W, p=float(wald_pvalue(W)), OR=OR,
                       ci_OR=(OR * math.exp(-1.96 * se_log), OR * math.exp(1.96 * se_log)))
    P = np.array([rows[k]["p"] for k in rows])
    return dict(placebo=p0, rows=rows, bonf=bonferroni(P, alpha), bh=bh_reject(P, alpha),
                bh_T=bh_threshold(P, alpha), by=bh_reject(P, alpha, True))


def ex12_sim(R=10000, n=20, lam0=1.0, alpha=0.05, seed=12):
    """Exercise 10.12(b): the type I error rate of the Poisson Wald test by simulation, and exactly."""
    rate, _ = wald_sim("poisson", lam0, lam0, n, alpha, R, seed)
    return dict(sim=rate, exact=wald_exact_power("poisson", lam0, lam0, n, alpha))


# =========================================================================== data for the demos
def export_json(path=None):
    """Write ../data/ch10-permutation.json (the four small data sets of Section 10.5 and Exercise 10.7)."""
    path = path or os.path.normpath(os.path.join(HERE, "..", "data", "ch10-permutation.json"))
    d = {"note": "Small two-sample data sets of Wasserman (2004), Chapter 10: the toy example 10.19, "
                 "genes 1 and 2 of Example 10.20 (log-ratio expression levels, five patients of each "
                 "type, from Efron et al. 2001) and the proportions of three-letter words of Exercise "
                 "10.7 (Brinegar 1963).",
         "sets": [
             {"key": "toy", "label": "Example 10.19 (toy)", "x": TOY[0].tolist(), "y": TOY[1].tolist(),
              "xlab": "X", "ylab": "Y"},
             {"key": "gene1", "label": "Example 10.20, gene 1", "x": GENE1[0].tolist(), "y": GENE1[1].tolist(),
              "xlab": "Type I", "ylab": "Type II"},
             {"key": "gene2", "label": "Example 10.20, gene 2", "x": GENE2[0].tolist(), "y": GENE2[1].tolist(),
              "xlab": "Type I", "ylab": "Type II"},
             {"key": "twain", "label": "Exercise 10.7, Twain v. Snodgrass", "x": TWAIN.tolist(),
              "y": SNODGRASS.tolist(), "xlab": "Twain", "ylab": "Snodgrass"}]}
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w") as f:
        json.dump(d, f, separators=(",", ":"))
    print("wrote", os.path.relpath(path, os.getcwd()))
    return path


# =========================================================================== figures of the book
def fig_10_1(out_dir=None):
    """Figure 10.1: the power function of the one-sided test of Example 10.2."""
    fig, axes = common.panel_grid(1, 2, size=(3.9, 2.9))
    ax = axes[0, 0]
    mu = np.linspace(-1.0, 1.5, 501)
    n, alpha = 10, 0.05
    b = power_normal(mu, n, 1.0, alpha)
    ax.plot(mu, b, color="k", lw=2)
    ax.axhline(alpha, color=BLUE, lw=0.9, ls="--")
    ax.axvline(0, color=GREY, lw=0.8)
    ax.plot([0], [alpha], "o", color=RED, ms=5)
    ax.text(-0.95, alpha + 0.03, r"$\alpha=%.2f$" % alpha, color=BLUE, fontsize=9)
    ax.text(-0.55, 0.45, r"$H_0:\ \mu\leq 0$", fontsize=9.5)
    ax.text(0.55, 0.45, r"$H_1:\ \mu>0$", fontsize=9.5)
    ax.set_xlabel(r"$\mu$")
    ax.set_ylabel(r"$\beta(\mu)$")
    ax.set_title(r"Example 10.2: $n=10$, $\sigma=1$, $c=\sigma z_\alpha/\sqrt{n}=%.3f$" % (zq(alpha) / math.sqrt(n)))
    ax.set_ylim(-0.02, 1.02)
    common.panel_label(ax, "a")

    ax = axes[0, 1]
    for nn, col in ((5, ORANGE), (20, BLUE), (80, PURPLE)):
        ax.plot(mu, power_normal(mu, nn, 1.0, 0.05), color=col, label="n = %d" % nn)
        ax.plot(mu, power_normal(mu, nn, 1.0, 0.01), color=col, ls="--", lw=1.1)
    ax.axvline(0, color=GREY, lw=0.8)
    ax.set_xlabel(r"$\mu$")
    ax.set_ylabel(r"$\beta(\mu)$")
    ax.set_title(r"solid $\alpha=0.05$, dashed $\alpha=0.01$")
    ax.legend(loc="center right", fontsize=8, frameon=False)
    ax.set_ylim(-0.02, 1.02)
    common.panel_label(ax, "b")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-1", out_dir)


def fig_10_4(out_dir=None):
    """Figure 10.4: the Wald p-value is the two-sided Normal tail area beyond |w|."""
    fig, axes = common.panel_grid(1, 2, size=(3.9, 2.7))
    z = np.linspace(-4.5, 4.5, 901)
    for ax, w, lab, title in ((axes[0, 0], 18.5 / 7.7, "a", "medians, Example 10.15: w = 2.40"),
                              (axes[0, 1], 20.92 / 5.55, "b", "means, Example 10.15: w = 3.77")):
        ax.plot(z, stats.norm.pdf(z), color="k", lw=1.6)
        for sgn in (-1, 1):
            zz = np.linspace(abs(w), 4.5, 200) * sgn
            ax.fill_between(zz, 0, stats.norm.pdf(zz), color=RED, alpha=0.55, lw=0)
            ax.axvline(sgn * abs(w), color=RED, lw=0.9, ymax=0.3)
        p = wald_pvalue(w)
        ax.text(0, 0.06, "p-value = 2$\\Phi(-|w|)$\n= %s" % ("%.4f" % p if p > 1e-3 else "%.5f" % p),
                ha="center", fontsize=8.5)
        ax.set_xticks([-abs(w), 0, abs(w)])
        ax.set_xticklabels([r"$-|w|$", "0", r"$|w|$"])
        ax.set_yticks([])
        ax.set_title(title)
        ax.set_ylim(0, 0.43)
        common.panel_label(ax, lab, loc="upper right")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-4", out_dir)


def fig_10_5(out_dir=None):
    """Figure 10.5: the p-value of a chi-squared test is the upper tail area beyond the observed t."""
    fig, axes = common.panel_grid(1, 2, size=(3.9, 2.7))
    v = np.linspace(1e-3, 14, 1000)
    T = float(pearson_stat(MENDEL_X, MENDEL_P0))
    for ax, t, lab, title in ((axes[0, 0], stats.chi2.isf(0.05, 3), "a", r"$t=\chi^2_{3,0.05}=7.815$: tail area 0.05"),
                              (axes[0, 1], T, "b", "Mendel's peas: t = %.2f" % T)):
        ax.plot(v, stats.chi2.pdf(v, 3), color="k", lw=1.6)
        vv = np.linspace(t, 14, 300)
        ax.fill_between(vv, 0, stats.chi2.pdf(vv, 3), color=BLUE, alpha=0.45, lw=0)
        ax.axvline(t, color=BLUE, lw=0.9, ymax=0.5)
        ax.text(0.97, 0.75, r"P$(\chi^2_3>t)$ = %.3f" % stats.chi2.sf(t, 3), transform=ax.transAxes,
                ha="right", fontsize=9)
        ax.set_xlabel("v")
        ax.set_ylabel(r"$\chi^2_3$ density")
        ax.set_title(title)
        ax.set_ylim(0, 0.3)
        common.panel_label(ax, lab, loc="upper left" if lab == "a" else "upper left")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-5", out_dir)


def _bh_panel(ax, p, alpha, title, logy=False):
    m = len(p)
    ps = np.sort(p)
    i = np.arange(1, m + 1)
    T = bh_threshold(p, alpha)
    rej = ps <= T if T > 0 else np.zeros(m, bool)
    ax.vlines(i, 0 if not logy else 1e-5, ps, color=GREY, lw=0.8, ls=":")
    ax.plot(i[rej], ps[rej], "o", color=RED, ms=5, label="rejected by BH")
    ax.plot(i[~rej], ps[~rej], "o", color="k", ms=5, mfc="white", label="not rejected")
    xx = np.linspace(0, m + 0.5, 50)
    ax.plot(xx, xx * alpha / m, color=BLUE, lw=1.4, label=r"line $i\alpha/m$")
    ax.axhline(alpha, color=GREEN, lw=1, ls="--", label=r"$\alpha$ (uncorrected)")
    ax.axhline(alpha / m, color=MAGENTA, lw=1, ls="-.", label=r"$\alpha/m$ (Bonferroni)")
    if T > 0:
        ax.axhline(T, color=RED, lw=0.8)
        ax.text(m + 0.4, T, " T", color=RED, va="center", fontsize=9)
    ax.set_xlabel("rank i of the p-value")
    ax.set_ylabel(r"$P_{(i)}$")
    ax.set_title(title)
    ax.set_xticks(i)
    if logy:
        ax.set_yscale("log")
        ax.set_ylim(1e-4, 1.5)
    ax.set_xlim(0.3, m + 0.9)


def fig_10_6(out_dir=None):
    """Figure 10.6: the Benjamini-Hochberg procedure (Examples 10.27 and 10.28)."""
    fig, axes = common.panel_grid(1, 2, size=(3.9, 3.0))
    ax = axes[0, 0]
    _bh_panel(ax, FIG_10_6_P, 0.05, "Example 10.27: six p-values")
    ax.set_ylim(0, 0.1)
    ax.legend(loc="upper left", fontsize=7, frameon=False)
    common.panel_label(ax, "a", loc="upper right")
    ax = axes[0, 1]
    _bh_panel(ax, EX_10_28, 0.05, "Example 10.28: ten p-values (log scale)", logy=True)
    common.panel_label(ax, "b", loc="lower right")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-6", out_dir)


# =========================================================================== supplementary figures
def fig_10_s1(out_dir=None):
    """10.S1: the power of the Wald test, exact against (10.6), and its exact size in discrete models."""
    fig, axes = common.panel_grid(1, 3, size=(3.5, 2.9))
    ax = axes[0, 0]
    th = np.linspace(-1, 1, 401)
    for n, col in ((10, ORANGE), (25, BLUE), (100, PURPLE)):
        ax.plot(th, wald_power_approx(th, 0.0, 1 / math.sqrt(n)), color=col, label="n = %d" % n)
    ax.axhline(0.05, color=GREY, lw=0.8, ls="--")
    ax.set_xlabel(r"true mean $\theta_\star$")
    ax.set_ylabel("power")
    ax.set_title(r"Normal mean, $H_0:\theta=0$, two-sided")
    ax.legend(loc="center right", fontsize=8, frameon=False)
    common.panel_label(ax, "a", loc="upper left")

    ax = axes[0, 1]
    n = 40
    ps = np.linspace(0.2, 0.8, 601)
    ex = [wald_exact_power("bernoulli", p, 0.5, n) for p in ps]
    ax.plot(ps, ex, color=RED, lw=1.3, label="exact")
    ax.plot(ps, wald_power_approx(ps, 0.5, true_se("bernoulli", ps, n)), color="k", ls="--", lw=1.1, label="(10.6)")
    ax.axhline(0.05, color=GREY, lw=0.8, ls=":")
    ax.set_xlabel(r"true $p$")
    ax.set_ylabel("power")
    ax.set_title(r"Bernoulli, $n=40$, $H_0: p=0.5$")
    ax.legend(loc="upper center", fontsize=8, frameon=False)
    common.panel_label(ax, "b", loc="upper left")

    ax = axes[0, 2]
    ns = np.arange(10, 201)
    for (model, t0, kind, col, lab) in (("bernoulli", 0.5, "hat", RED, r"Bernoulli $p_0=0.5$"),
                                        ("bernoulli", 0.1, "hat", ORANGE, r"Bernoulli $p_0=0.1$"),
                                        ("bernoulli", 0.1, "null", MAGENTA, r"$p_0=0.1$, se at null"),
                                        ("poisson", 1.0, "hat", BLUE, r"Poisson $\lambda_0=1$")):
        ax.plot(ns, [wald_exact_power(model, t0, t0, n_, 0.05, kind) for n_ in ns], color=col, lw=1.0, label=lab)
    ax.axhline(0.05, color="k", lw=0.8, ls="--")
    ax.set_xlabel("sample size n")
    ax.set_ylabel("exact size")
    ax.set_title(r"exact size of the 5% Wald test")
    ax.set_ylim(0, 0.16)
    ax.legend(loc="upper right", fontsize=7, frameon=False)
    common.panel_label(ax, "c", loc="upper left")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-S1", out_dir)


def fig_10_s2(out_dir=None):
    """10.S2: the distribution of the p-value under H0 and H1 (Theorem 10.14), discrete null
    distributions, and P(H0 | reject)."""
    fig, axes = common.panel_grid(1, 3, size=(3.5, 2.9))
    ax = axes[0, 0]
    rng = np.random.default_rng(1014)
    for delta, col, lab in ((0.0, BLUE, r"$H_0$ true"), (1.5, ORANGE, r"$\delta=1.5$"), (3.0, RED, r"$\delta=3$")):
        p = wald_pvalue(rng.standard_normal(20000) + delta)
        ax.hist(p, bins=40, range=(0, 1), density=True, histtype="step", color=col, lw=1.0)
        u = np.linspace(0.001, 0.999, 300)
        ax.plot(u, pvalue_pdf_wald(u, delta), color=col, lw=1.6, alpha=0.7, label=lab)
    ax.set_ylim(0, 6)
    ax.set_xlabel("p-value")
    ax.set_ylabel("density")
    ax.set_title("Wald p-values: uniform under $H_0$")
    ax.legend(loc="center right", fontsize=8, frameon=False)
    common.panel_label(ax, "a", loc="upper right")

    ax = axes[0, 1]
    T = perm_distribution(GENE1[0], GENE1[1], "median")
    pge = np.array([perm_pvalues(T, t)[0] for t in T])
    pgt = np.array([perm_pvalues(T, t)[1] for t in T])
    u = np.linspace(0, 1, 1001)
    ax.plot(u, [np.mean(pge <= a) for a in u], color=BLUE, lw=1.2, label=r"$P_0(T\geq t)$")
    ax.plot(u, [np.mean(pgt <= a) for a in u], color=RED, lw=1.2, label=r"$P_0(T>t)$")
    ax.plot([0, 1], [0, 1], color="k", lw=0.8, ls="--")
    ax.set_xlim(0, 0.3)
    ax.set_ylim(0, 0.45)
    ax.set_xlabel(r"$u$")
    ax.set_ylabel(r"$P_{H_0}(\mathrm{p\text{-}value}\leq u)$")
    ax.set_title("permutation p-values, gene 1, medians")
    ax.legend(loc="upper left", fontsize=8, frameon=False)
    common.panel_label(ax, "b", loc="lower right")

    ax = axes[0, 2]
    pi0 = np.linspace(0, 0.99, 300)
    for pw, col in ((0.8, BLUE), (0.5, ORANGE), (0.2, RED)):
        ax.plot(pi0, posterior_null(pi0, 0.05, pw), color=col, label="power %.1f" % pw)
    ax.plot([0, 1], [0, 1], color="k", lw=0.8, ls="--")
    ax.set_xlabel(r"fraction of true nulls $\pi_0$")
    ax.set_ylabel(r"P($H_0$ | reject at 5%)")
    ax.set_title("a rejection is not P(H0) = 0.05")
    ax.legend(loc="upper left", fontsize=8, frameon=False)
    common.panel_label(ax, "c", loc="lower right")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-S2", out_dir)


def fig_10_s3(out_dir=None):
    """10.S3: the chi-squared family; Pearson and LRT for Mendel's peas; exact size of LRTs."""
    fig, axes = common.panel_grid(1, 3, size=(3.5, 2.9))
    ax = axes[0, 0]
    v = np.linspace(0.01, 20, 800)
    for k, col in ((1, RED), (2, ORANGE), (3, GREEN), (5, BLUE), (10, PURPLE)):
        ax.plot(v, stats.chi2.pdf(v, k), color=col, label="k = %d" % k)
    ax.set_ylim(0, 0.5)
    ax.set_xlabel("v")
    ax.set_ylabel("density")
    ax.set_title(r"$\chi^2_k$: mean $k$, variance $2k$")
    ax.legend(loc="center right", fontsize=8, frameon=False)
    common.panel_label(ax, "a", loc="upper right")

    ax = axes[0, 1]
    Tp, Tl = mendel_null_sim()
    q = np.linspace(0.0025, 0.9975, 399)
    th = stats.chi2.ppf(q, 3)
    ax.plot(th, np.quantile(Tp, q), ".", color=BLUE, ms=2.5, label="Pearson T")
    ax.plot(th, np.quantile(Tl, q), ".", color=RED, ms=2.5, label=r"LRT $\lambda$")
    ax.plot([0, 16], [0, 16], color="k", lw=0.8)
    T0 = float(pearson_stat(MENDEL_X, MENDEL_P0))
    ax.axhline(T0, color=GREY, lw=0.8, ls=":")
    ax.text(15.5, T0 + 0.4, "observed 0.47", ha="right", fontsize=7.5, color=GREY)
    ax.set_xlabel(r"$\chi^2_3$ quantile")
    ax.set_ylabel("simulated quantile")
    ax.set_title("Mendel: 20000 samples of n = 556")
    ax.legend(loc="upper left", fontsize=8, frameon=False)
    common.panel_label(ax, "b", loc="upper right")

    ax = axes[0, 2]
    ns = np.arange(3, 61)
    ax.plot(ns, [lrt_exact_size("mean", n) for n in ns], color=BLUE, label="Ex. 10.13 (mean)")
    ax.plot(ns, [lrt_exact_size("sd", n) for n in ns], color=RED, label="Ex. 10.14 (sd)")
    ax.axhline(0.05, color="k", lw=0.8, ls="--")
    ax.set_xlabel("sample size n")
    ax.set_ylabel(r"exact size with $\chi^2_{1,0.05}$")
    ax.set_title("Wilks' limit in small samples")
    ax.set_ylim(0, 0.25)
    ax.legend(loc="upper right", fontsize=8, frameon=False)
    common.panel_label(ax, "c", loc="upper left")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-S3", out_dir)


def fig_10_s4(out_dir=None):
    """10.S4: exact permutation distributions (Example 10.20, Exercise 10.7) and sizes in small samples."""
    fig, axes = common.panel_grid(1, 3, size=(3.5, 2.9))
    ax = axes[0, 0]
    T = perm_distribution(GENE1[0], GENE1[1], "median")
    vals, cnt = np.unique(np.round(T, 6), return_counts=True)
    t0 = T[0]
    ax.vlines(vals, 0, cnt / len(T), color=np.where(vals >= t0 - 1e-6, RED, BLUE), lw=2)
    pge, pgt = perm_pvalues(T, t0)
    ax.set_xlabel("|difference of medians|")
    ax.set_ylabel("permutation probability")
    ax.set_title("gene 1: all 252 splits, t = 710")
    ax.text(0.97, 0.80, "P(T >= 710) = %.3f\nP(T > 710) = %.3f" % (pge, pgt), transform=ax.transAxes,
            ha="right", va="top", fontsize=8)
    common.panel_label(ax, "a", loc="upper right")

    ax = axes[0, 1]
    T = perm_distribution(TWAIN, SNODGRASS, "mean")
    t0 = T[0]
    ax.hist(T, bins=60, color=BLUE, alpha=0.75)
    ax.axvline(t0, color=RED, lw=1.2)
    pge, _ = perm_pvalues(T, t0)
    ax.set_xlabel("|difference of means|")
    ax.set_ylabel("number of splits")
    ax.set_title("Twain v. Snodgrass: 43758 splits")
    ax.text(0.97, 0.95, "observed %.4f\np = %d/43758\n= %.5f" % (t0, round(pge * len(T)), pge),
            transform=ax.transAxes, ha="right", va="top", fontsize=8)
    common.panel_label(ax, "b", loc="center right")

    ax = axes[0, 2]
    labs = ("Wald", "pooled t", "perm. (>=)", "perm. (>)")
    keys = ("wald", "t", "perm", "perm_gt")
    cols = (ORANGE, GREEN, BLUE, RED)
    dists = ("normal", "lognormal", "unequal")
    width = 0.2
    for j, d in enumerate(dists):
        m, n = (5, 5) if d != "unequal" else (6, 3)
        r = perm_size_sim(d, m, n, 0.05, 2000, 7)
        for i, k in enumerate(keys):
            ax.bar(j + (i - 1.5) * width, r[k], width, color=cols[i], label=labs[i] if j == 0 else None)
    ax.axhline(0.05, color="k", lw=0.8, ls="--")
    ax.set_ylim(0, 0.27)
    ax.set_xticks(range(3))
    ax.set_xticklabels(["Normal\nm=n=5", "lognormal\nm=n=5", "sd 1 v. 3\nm=6, n=3"], fontsize=8)
    ax.set_ylabel("rejection rate under H0")
    ax.set_title("size of 5% tests, 2000 data sets")
    ax.legend(loc="upper left", fontsize=7, frameon=False)
    fig.tight_layout()
    common.panel_label(ax, "c", loc="upper right")
    return savefig(fig, CH, "fig10-S4", out_dir)


def fig_10_s5(out_dir=None):
    """10.S5: FWER and FDR of uncorrected testing, Bonferroni and BH (Theorems 10.24 and 10.26)."""
    m = 50
    m0s = np.array([5, 10, 20, 30, 40, 45, 50])
    res = {k: {"fwer": [], "fdr": []} for k in ("none", "bonferroni", "bh", "by")}
    res_dep = {"bh": []}
    for m0 in m0s:
        r = mt_sim(m, int(m0), 3.0, 0.0, 0.05, 2000, 100 + int(m0))
        for k in res:
            res[k]["fwer"].append(r[k]["fwer"])
            res[k]["fdr"].append(r[k]["fdr"])
        rd = mt_sim(m, int(m0), 3.0, 0.6, 0.05, 2000, 200 + int(m0))
        res_dep["bh"].append(rd["bh"]["fdr"])
    fig, axes = common.panel_grid(1, 2, size=(3.9, 2.9))
    cols = {"none": GREEN, "bonferroni": MAGENTA, "bh": BLUE, "by": ORANGE}
    labs = {"none": "uncorrected", "bonferroni": "Bonferroni", "bh": "BH", "by": r"BH with $C_m$"}
    frac = m0s / m
    for ax, key, lab in ((axes[0, 0], "fwer", "a"), (axes[0, 1], "fdr", "b")):
        for k in res:
            ax.plot(frac, res[k][key], "o-", color=cols[k], ms=3.5, lw=1.1, label=labs[k])
        ax.axhline(0.05, color="k", lw=0.8, ls="--")
        ax.set_xlabel(r"fraction of true nulls $m_0/m$")
        ax.set_ylabel(key.upper())
        common.panel_label(ax, lab, loc="upper right" if key == "fdr" else "upper left")
    axes[0, 1].plot(frac, frac * 0.05, color=GREY, lw=1, ls=":", label=r"$m_0\alpha/m$")
    axes[0, 1].plot(frac, res_dep["bh"], "s--", color=PURPLE, ms=3.5, lw=1, label=r"BH, $\rho=0.6$")
    axes[0, 0].set_title("family-wise error rate, m = 50")
    axes[0, 1].set_title("false discovery rate, m = 50")
    axes[0, 0].set_yscale("log")
    axes[0, 0].set_ylim(3e-4, 3)
    axes[0, 1].set_ylim(0, 0.3)
    axes[0, 0].legend(loc="lower center", fontsize=7, frameon=False, ncol=2)
    axes[0, 1].legend(loc="upper left", fontsize=7, frameon=False)
    fig.tight_layout()
    return savefig(fig, CH, "fig10-S5", out_dir)


def fig_10_s6(out_dir=None):
    """10.S6: the Neyman-Pearson lemma: ROC curves of most powerful tests and of competitors, and the
    power function of Exercise 10.5."""
    fig, axes = common.panel_grid(1, 3, size=(3.5, 2.9))
    for ax, fam, n, delta, lab, title, cname in (
            (axes[0, 0], "normal", 10, 0.5, "a", r"N(0,1) v. N(0.5,1), n = 10", "median"),
            (axes[0, 1], "cauchy", 5, 1.0, "b", "Cauchy(0) v. Cauchy(1), n = 5", "median")):
        r = np_mc(fam, n, delta, 0.05, 5000, 3)
        s, pw = roc_points(r["llr0"], r["llr1"])
        ax.plot(s, pw, color=RED, lw=1.5, label="likelihood ratio (NP)")
        s, pw = roc_points(r["comp0"], r["comp1"])
        ax.plot(s, pw, color=BLUE, lw=1.2, label=cname)
        if fam == "normal":
            a = np.linspace(1e-4, 1, 300)
            ax.plot(a, stats.norm.sf(stats.norm.isf(a) - math.sqrt(n) * delta), color="k", lw=0.8, ls="--", label="exact NP")
        ax.plot([0, 1], [0, 1], color=GREY, lw=0.7, ls=":")
        ax.axvline(0.05, color=GREY, lw=0.7)
        ax.set_xlabel(r"size $\alpha$")
        ax.set_ylabel(r"power $\beta(\theta_1)$")
        ax.set_title(title)
        ax.legend(loc="lower right", fontsize=7.5, frameon=False)
        common.panel_label(ax, lab, loc="upper left")
    ax = axes[0, 2]
    th = np.linspace(0.3, 0.8, 500)
    for n, col in ((5, ORANGE), (20, BLUE), (80, PURPLE)):
        c = ex5(n)["c"]
        ax.plot(th, np.where(th > c, 1 - (c / th) ** n, 0.0), color=col, label="n = %d, c = %.4f" % (n, c))
    ax.axhline(0.05, color="k", lw=0.8, ls="--")
    ax.axvline(0.5, color=GREY, lw=0.8)
    ax.set_xlabel(r"$\theta$")
    ax.set_ylabel(r"$\beta(\theta)$")
    ax.set_title(r"Exercise 10.5: Uniform$(0,\theta)$, $Y>c$")
    ax.legend(loc="lower right", fontsize=7.5, frameon=False)
    common.panel_label(ax, "c", loc="upper left")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-S6", out_dir)


def fig_10_s7(out_dir=None):
    """10.S7: a goodness-of-fit test of the Exponential model for the nerve data (Theorem 10.29)."""
    g = gof_nerve()
    fig, axes = common.panel_grid(1, 2, size=(3.9, 2.9))
    ax = axes[0, 0]
    k = g["k"]
    E = len(g["x"]) * exp_cell_probs(g["rate_grouped"], NERVE_EDGES)
    idx = np.arange(k)
    ax.bar(idx - 0.2, g["counts"], 0.4, color=BLUE, label="observed $N_j$")
    ax.bar(idx + 0.2, E, 0.4, color=ORANGE, label=r"expected $np_j(\tilde\theta)$")
    labels = ["%.3g" % e for e in NERVE_EDGES[:-1]]
    ax.set_xticks(idx)
    ax.set_xticklabels(labels, rotation=60, fontsize=7)
    ax.set_xlabel("left end of the cell (seconds)")
    ax.set_ylabel("count")
    ax.set_title(r"nerve data: $Q=%.1f$, p = %.3f ($\chi^2_{%d}$)" % (g["Q_grouped"], g["p_grouped"], k - 2))
    ax.set_ylim(0, 150)
    ax.legend(loc="upper left", fontsize=7.5, frameon=False, ncol=2)
    common.panel_label(ax, "a", loc="upper right")

    ax = axes[0, 1]
    Qg, Qr = gof_null_sim(len(g["x"]), g["rate_grouped"], 2000)
    q = np.linspace(0.005, 0.995, 199)
    th = stats.chi2.ppf(q, k - 2)
    ax.plot(th, np.quantile(Qg, q), ".", color=BLUE, ms=3, label="grouped MLE")
    ax.plot(th, np.quantile(Qr, q), ".", color=RED, ms=3, label="raw-data MLE")
    ax.plot(th, stats.chi2.ppf(q, k - 1), color=GREY, lw=1, ls="--", label=r"$\chi^2_{%d}$ quantiles" % (k - 1))
    ax.plot([0, th.max()], [0, th.max()], color="k", lw=0.8)
    ax.set_xlabel(r"$\chi^2_{%d}$ quantile" % (k - 2))
    ax.set_ylabel("simulated quantile of Q")
    ax.set_title("null distribution of Q (2000 samples)")
    ax.legend(loc="upper left", fontsize=7.5, frameon=False)
    common.panel_label(ax, "b", loc="lower right")
    fig.tight_layout()
    return savefig(fig, CH, "fig10-S7", out_dir)


FIGURES = {"10.1": fig_10_1, "10.4": fig_10_4, "10.5": fig_10_5, "10.6": fig_10_6,
           "10.S1": fig_10_s1, "10.S2": fig_10_s2, "10.S3": fig_10_s3, "10.S4": fig_10_s4,
           "10.S5": fig_10_s5, "10.S6": fig_10_s6, "10.S7": fig_10_s7}

if __name__ == "__main__":
    export_json()
    common.run(FIGURES, CH)
