"""Wasserman (2006), Chapter 5: Nonparametric Regression.

Companion code for Chapter 5 of L. Wasserman, All of Nonparametric Statistics (Springer, 2006),
pp. 61-123, written from scratch with numpy, scipy and matplotlib, to be read next to the lecture
notes (../ch05-nonparametric-regression.html). Everything the chapter is about is implemented here;
library versions (scipy, scikit-learn) are only used to confirm the implementations.

Methods
* Section 5.1: ols_fit (least squares and the hat matrix (5.6)-(5.11)), scheffe_band (Theorem 5.14),
  logistic_irls (the reweighted least squares algorithm).
* Sections 5.2-5.4, linear smoothers: kernel (tricube, Gaussian, Epanechnikov, boxcar),
  locpoly_matrix (local polynomial weights of any degree p, Theorem 5.57, and the derivative weights
  (5.143)), nw_matrix and ll_matrix (the closed forms (5.41) and (5.61)), regressogram_matrix
  (Example 5.24), loo_cv (the shortcut (5.35)), loo_brute (refitting with the renormalised weights
  (5.32)), gcv (5.36), cp_score (5.37), select_h (a bandwidth search).
* Section 5.5, splines: bspline_basis (the Cox-de Boor recursion of the book), bspline_d2 (second
  derivatives), omega_matrix (the penalty matrix of (5.78)), smoothing_spline (Theorems 5.73-5.81),
  spline_df_to_lam, regression_spline, truncated_power_basis (Theorem 5.74), silverman_kernel.
* Section 5.6: sigma2_linear (5.86), sigma2_rice (5.89), sigma2_gasser (5.91), variance_function (5.95).
* Section 5.7: tube_kappa0 (5.101), tube_c (5.102, Normal and t versions), linear_band (5.99), (5.103),
  (5.104).
* Section 5.10: local_logistic (local linear likelihood, (5.115)), local_logistic_cv (5.116), by
  refitting n times, and its approximation (5.117)-(5.118).
* Section 5.12: loclin_nd (5.122)-(5.124), backfit (the backfitting algorithm), ppr (projection pursuit
  regression, Friedman and Stuetzle 1981), tree_fit (CART with cost-complexity pruning, via
  scikit-learn), mars (a forward/backward MARS fit).
* Section 5.13: plugin_h (5.138), linearity_test, robust_local (Huber and quantile local linear fits).

Data. The CMB data are the 899 WMAP power-spectrum points of wmap.dat (multipoles 2-900) on the
book's website; the first 400 are used where the book uses them. LIDAR (lidar.dat), BPD (bpd.dat),
rock (rock.dat), glass (glass.dat) and motorcycle (motor.dat) data come from the same website.
The air quality data of Exercises 14 and 22 are not on the website; they are R's datasets::airquality
(New York, May-September 1973), exported once with Rscript into cache/airquality.csv.

The rock data. In the book's Figures 5.14, 5.15 and 5.19 the axes labelled area, perimeter and shape
cover the ranges of the columns peri, shape and perm, and the response labelled log permeability has
the range of log(area): the columns were read shifted by one. Here the stated model is fitted: log
permeability on area, perimeter and shape (see the notes, Notes on the book).

Figures. Figure 5.18 is a schematic of a two-split tree with no data and is skipped. Every other
figure, 5.1-5.17 and 5.19, is reproduced. Supplementary figures for the gaps filled in the notes:
  5.S1 design bias and boundary bias of the kernel and local linear estimators (Theorems 5.44, 5.65);
  5.S2 the smoothing spline is close to a kernel smoother (Silverman's equivalent kernel);
  5.S3 the tube formula against simulation: the coverage of the band for E r_n and for r (Section 5.7);
  5.S4 and 5.S5 Exercise 3 (glass): four smoothers with their bands, and their effective kernels;
  5.S6 Exercise 10: B-spline bases of order 1 to 5 on five knots;
  5.S7 Exercises 4 and 11 (motorcycle): local linear, regression spline and smoothing spline;
  5.S8 Exercises 12 and 13: the Doppler function at three noise levels;
  5.S9 Exercise 14: ozone on temperature, four bandwidth selectors;
  5.S10 Exercise 23: the additive model for ozone with bands for its components (Exercise 22 is
        ozone_compare, a table);
  5.S11 Exercise 24: the null distribution of the test for dropping a covariate;
  5.S12 Exercise 9 and Section 5.11: fits across bandwidths and a SiZer map (glass);
  5.S13 Exercise 18: local median and Huber regression (glass).
Run `python3 ch05_nonparametric_regression.py --list`, `python3 ch05_nonparametric_regression.py` (all
figures, about two minutes on one CPU thread) or `python3 ch05_nonparametric_regression.py --fig 5.4`.
Files go to ../img/ch05/; the data used by the browser demonstrations go to ../data/ch05-*.json.
"""
import json
import math
import os
import subprocess

import numpy as np
from scipy import linalg, optimize, stats

import common
from common import BLUE, GREEN, GREY, MAGENTA, ORANGE, PURPLE, RED, savefig

CH = 5
HERE = os.path.dirname(os.path.abspath(__file__))
DATA_OUT = os.path.normpath(os.path.join(HERE, "..", "data"))
CACHE_DIR = os.path.join(HERE, "cache")
M32 = 0xFFFFFFFF


# =========================================================================== the demos' generator
def _imul(a, b):
    return (a * b) & M32


class JSRandom:
    """The seeded generator of the browser demos (DK.rng in courses/demo-kit.js): u() is mulberry32
    and norm() the Box-Muller transform that returns r cos(t) and keeps r sin(t) for the next call."""

    def __init__(self, seed):
        self.a = int(seed) & M32
        self.spare = None

    def u(self):
        a = (self.a + 0x6D2B79F5) & M32
        self.a = a
        t = _imul(a ^ (a >> 15), 1 | a)
        t = ((t + _imul(t ^ (t >> 7), 61 | t)) & M32) ^ t
        return ((t ^ (t >> 14)) & M32) / 4294967296.0

    def norm(self):
        if self.spare is not None:
            s, self.spare = self.spare, None
            return s
        x = 0.0
        while x == 0.0:
            x = self.u()
        r = math.sqrt(-2.0 * math.log(x))
        th = 2.0 * math.pi * self.u()
        self.spare = r * math.sin(th)
        return r * math.cos(th)


# =========================================================================== data
def cmb(n=None):
    """WMAP CMB power spectrum (wmap.dat): multipole l, power C_l and the standard error of C_l
    computed by the WMAP team (the 'known' sigma(x) of Example 5.96). n keeps the first n points."""
    d = common.nps_data("wmap.dat")
    x, y, se = (d[c].to_numpy(float) for c in ("ell", "Cl", "se"))
    if n is not None:
        x, y, se = x[:n], y[:n], se[:n]
    return x, y, se


def lidar():
    """LIDAR data (lidar.dat): range (distance travelled by the light) and log ratio, n = 221."""
    d = common.nps_data("lidar.dat")
    return d["range"].to_numpy(float), d["logratio"].to_numpy(float)


def bpd():
    """BPD data (bpd.dat): birth weight (grams) and bronchopulmonary dysplasia (0/1), n = 223."""
    d = common.nps_data("bpd.dat")
    return d["birthweight"].to_numpy(float), d["BPD"].to_numpy(float)


def rock():
    """Rock data (rock.dat): 48 rock samples; X = (area, peri, shape), Y = log(perm)."""
    d = common.nps_data("rock.dat")
    X = d[["area", "peri", "shape"]].to_numpy(float)
    return X, np.log(d["perm"].to_numpy(float))


def glass():
    """Glass fragments (glass.dat): aluminium content Al (x) and refractive index RI (Y), n = 214."""
    d = common.nps_data("glass.dat")
    return d["Al"].to_numpy(float), d["RI"].to_numpy(float)


def motor():
    """Motorcycle data (motor.dat): time after impact (ms) and head acceleration (g), n = 94."""
    d = common.nps_data("motor.dat")
    return d["times"].to_numpy(float), d["accel"].to_numpy(float)


def airquality():
    """R's datasets::airquality, complete cases (n = 111): columns Ozone, Solar.R, Wind, Temp."""
    path = os.path.join(CACHE_DIR, "airquality.csv")
    if not os.path.exists(path):
        os.makedirs(CACHE_DIR, exist_ok=True)
        subprocess.run(["/usr/local/bin/Rscript", "-e",
                        "write.csv(na.omit(datasets::airquality)[,1:4], '%s', row.names=FALSE)" % path], check=True)
    import pandas as pd
    return pd.read_csv(path)


def doppler(x):
    """The Doppler function (5.64)."""
    return np.sqrt(x * (1 - x)) * np.sin(2.1 * np.pi / (x + 0.05))


def bumps4(x):
    """The four-bump function of Example 5.98: Normal densities with sds 1, 0.5, 0.1, 0.05."""
    p = stats.norm.pdf
    return p(x, 2, 1) + p(x, 4, 0.5) + p(x, 6, 0.1) + p(x, 8, 0.05)


# =========================================================================== Section 5.1
def ols_fit(X, Y):
    """Least squares (5.6): beta, the hat matrix L (5.8), the residuals and sigma2 (5.11)."""
    beta = np.linalg.lstsq(X, Y, rcond=None)[0]
    L = X @ np.linalg.solve(X.T @ X, X.T)
    res = Y - X @ beta
    return beta, L, res, res @ res / (len(Y) - X.shape[1])


def scheffe_band(X, Y, Xg, alpha=0.05):
    """The band (5.13) with c = sqrt(p F_{alpha; p, n-p}) of Theorem 5.14, at the rows of Xg."""
    n, p = X.shape
    beta, _, _, s2 = ols_fit(X, Y)
    A = np.linalg.solve(X.T @ X, Xg.T)              # columns (X^T X)^{-1} x
    norm_l = np.sqrt(np.einsum("ij,ji->i", Xg, A))   # ||l(x)|| = sqrt(x^T (X^T X)^{-1} x)
    c = math.sqrt(p * stats.f.ppf(1 - alpha, p, n - p))
    f = Xg @ beta
    return f, f - c * math.sqrt(s2) * norm_l, f + c * math.sqrt(s2) * norm_l, c


def logistic_irls(X, Y, iters=50, tol=1e-10):
    """Reweighted least squares for logistic regression (Section 5.1)."""
    beta = np.zeros(X.shape[1])
    for _ in range(iters):
        eta = X @ beta
        p = 1 / (1 + np.exp(-eta))
        w = np.clip(p * (1 - p), 1e-12, None)
        Z = eta + (Y - p) / w
        new = np.linalg.solve(X.T @ (w[:, None] * X), X.T @ (w * Z))
        if np.max(np.abs(new - beta)) < tol:
            beta = new
            break
        beta = new
    return beta


# =========================================================================== Sections 5.2-5.4: linear smoothers
def kernel(u, kind="tricube"):
    """Kernels on the real line: tricube (the book's default), gauss, epan (Epanechnikov), boxcar."""
    u = np.asarray(u, dtype=float)
    a = np.abs(u)
    if kind == "tricube":
        return np.where(a < 1, 70.0 / 81.0 * (1 - a ** 3) ** 3, 0.0)
    if kind == "gauss":
        return np.exp(-0.5 * u * u) / math.sqrt(2 * math.pi)
    if kind == "epan":
        return np.where(a < 1, 0.75 * (1 - u * u), 0.0)
    if kind == "boxcar":
        return np.where(a <= 1, 0.5, 0.0)
    raise ValueError(kind)


def nw_matrix(x, grid, h, kind="tricube"):
    """Nadaraya-Watson weights (5.41): row k holds l(grid_k)."""
    W = kernel((np.asarray(grid)[:, None] - np.asarray(x)[None, :]) / h, kind)
    with np.errstate(invalid="ignore", divide="ignore"):
        return W / W.sum(1, keepdims=True)


def ll_matrix(x, grid, h, kind="tricube"):
    """Local linear weights in the closed form (5.61) of Theorem 5.60."""
    D = np.asarray(x)[None, :] - np.asarray(grid)[:, None]          # x_i - x
    W = kernel(D / h, kind)
    S1 = (W * D).sum(1, keepdims=True)
    S2 = (W * D * D).sum(1, keepdims=True)
    B = W * (S2 - D * S1)
    with np.errstate(invalid="ignore", divide="ignore"):
        return B / B.sum(1, keepdims=True)


def locpoly_matrix(x, grid, h, p=1, kind="tricube", deriv=0):
    """Local polynomial weights of degree p (Theorem 5.57): row k is e_{deriv+1}^T (X^T W X)^{-1} X^T W
    at grid_k, with the columns (x_i - x)^j / j! of (5.54); deriv > 0 gives the derivative weights
    l(x, k) of (5.143). Computed in the scaled variable u = (x_i - x)/h for numerical stability."""
    x = np.asarray(x, float)
    grid = np.atleast_1d(np.asarray(grid, float))
    U = (x[None, :] - grid[:, None]) / h
    W = kernel(U, kind)
    P = np.stack([U ** j for j in range(p + 1)], axis=2)               # (G, n, p+1)
    M = np.einsum("gn,gnj,gnk->gjk", W, P, P)
    e = np.zeros(p + 1)
    e[deriv] = 1.0
    out = np.full((len(grid), len(x)), np.nan)
    for g in range(len(grid)):
        try:
            c = np.linalg.solve(M[g], e)
        except np.linalg.LinAlgError:
            continue
        if not np.all(np.isfinite(c)) or np.linalg.cond(M[g]) > 1e12:
            continue
        out[g] = W[g] * (P[g] @ c)
    # with u = (x_i - x)/h the coefficient of u^k is a_k h^k / k!, so a_k = k! b_k / h^k
    return out * math.factorial(deriv) / h ** deriv


def regressogram_matrix(x, a, b, m):
    """Smoothing matrix of the regressogram (5.25) with m equal bins on (a, b]."""
    j = np.clip(np.floor((np.asarray(x) - a) / (b - a) * m).astype(int), 0, m - 1)
    S = (j[:, None] == j[None, :]).astype(float)
    return S / S.sum(1, keepdims=True)


def smoother_matrix(x, h, method="ll", kind="tricube", p=None):
    """n x n smoothing matrix of a local smoother at the design points."""
    if method == "nw":
        return nw_matrix(x, x, h, kind)
    if method == "ll":
        return ll_matrix(x, x, h, kind)
    return locpoly_matrix(x, x, h, p, kind)


def loo_cv(L, y):
    """Leave-one-out cross-validation score by the shortcut (5.35) (inf if a weight L_ii = 1)."""
    r = y - L @ y
    d = 1 - np.diag(L)
    if not np.all(np.isfinite(r)) or np.any(d < 1e-10):
        return np.inf
    return float(np.mean((r / d) ** 2))


def loo_brute(Lfun, x, y):
    """Leave-one-out score by refitting with observation i removed. Lfun(xtrain, xeval) returns the
    weights at xeval computed from xtrain only."""
    n = len(y)
    s = 0.0
    for i in range(n):
        keep = np.arange(n) != i
        w = Lfun(x[keep], np.array([x[i]]))[0]
        s += (y[i] - w @ y[keep]) ** 2
    return s / n


def gcv(L, y):
    """Generalised cross-validation (5.36)."""
    nu = np.trace(L)
    n = len(y)
    r = y - L @ y
    return float(np.mean(r ** 2) / (1 - nu / n) ** 2)


def cp_score(L, y, s2=None):
    """The Cp form (5.37): training error + 2 nu s2 / n; s2 defaults to the training error, as in (5.37)."""
    r = y - L @ y
    n = len(y)
    if s2 is None:
        s2 = np.mean(r ** 2)
    return float(np.mean(r ** 2) + 2 * np.trace(L) * s2 / n)


def log_grid(lo, hi, k):
    return np.exp(np.linspace(math.log(lo), math.log(hi), k))


def select_h(x, y, hs, method="ll", kind="tricube", p=None, score="cv"):
    """Scores over the bandwidths hs: dict with h, cv, gcv, df, and the minimiser."""
    out = {"h": np.asarray(hs, float), "cv": [], "gcv": [], "df": []}
    for h in hs:
        L = smoother_matrix(x, h, method, kind, p)
        if not np.all(np.isfinite(L)):
            out["cv"].append(np.inf); out["gcv"].append(np.inf); out["df"].append(np.nan)
            continue
        out["cv"].append(loo_cv(L, y)); out["gcv"].append(gcv(L, y)); out["df"].append(np.trace(L))
    for k in ("cv", "gcv", "df"):
        out[k] = np.array(out[k])
    out["best"] = int(np.argmin(out[score]))
    out["hbest"] = float(out["h"][out["best"]])
    return out


# =========================================================================== Section 5.5: splines
def augmented_knots(interior, a, b, M=4):
    """tau_1..tau_{k+2M}: M copies of a, the interior knots, M copies of b (the usual choice)."""
    return np.concatenate([np.full(M, a), np.sort(interior), np.full(M, b)])


def bspline_basis(x, tau, M=4):
    """B_{i,M}(x), i = 1..k+M, by the recursion of Section 5.5 (0/0 taken as 0). The first-order
    functions are indicators of [tau_i, tau_{i+1}); the right end point b is put in the last
    non-empty interval so that the basis is a partition of unity on all of [a, b]."""
    x = np.asarray(x, float)
    t = np.asarray(tau, float)
    nb1 = len(t) - 1
    B = np.zeros((len(x), nb1))
    for i in range(nb1):
        B[:, i] = (t[i] <= x) & (x < t[i + 1])
    last = np.max(np.nonzero(t[:-1] < t[1:])[0])
    B[x == t[-1], last] = 1.0
    for m in range(2, M + 1):
        Bn = np.zeros((len(x), nb1 - m + 1))
        for i in range(nb1 - m + 1):
            d1 = t[i + m - 1] - t[i]
            d2 = t[i + m] - t[i + 1]
            a1 = (x - t[i]) / d1 * B[:, i] if d1 > 0 else 0.0
            a2 = (t[i + m] - x) / d2 * B[:, i + 1] if d2 > 0 else 0.0
            Bn[:, i] = a1 + a2
        B = Bn
    return B


def bspline_deriv(x, tau, M=4, order=1):
    """Derivatives of B_{i,M} by B'_{i,m} = (m-1)[B_{i,m-1}/(t_{i+m-1}-t_i) - B_{i+1,m-1}/(t_{i+m}-t_{i+1})]."""
    if order == 0:
        return bspline_basis(x, tau, M)
    t = np.asarray(tau, float)
    Bl = bspline_deriv(x, tau, M - 1, order - 1)       # derivatives of order-1 of the B_{i,M-1}
    nbM = len(t) - M
    D = np.zeros((len(np.atleast_1d(x)), nbM))
    for i in range(nbM):
        d1 = t[i + M - 1] - t[i]
        d2 = t[i + M] - t[i + 1]
        if d1 > 0:
            D[:, i] += (M - 1) * Bl[:, i] / d1
        if d2 > 0:
            D[:, i] -= (M - 1) * Bl[:, i + 1] / d2
    return D


def omega_matrix(tau, M=4):
    """Omega_jk = int B_j'' B_k'' over [a, b] (5.78). For cubic B-splines B'' is linear on each knot
    interval, so the two-point Gauss-Legendre rule on each interval is exact."""
    t = np.asarray(tau, float)
    br = np.unique(t)
    g = np.array([-1, 1]) / math.sqrt(3)
    xs, ws = [], []
    for lo, hi in zip(br[:-1], br[1:]):
        xs.append((lo + hi) / 2 + (hi - lo) / 2 * g)
        ws.append(np.full(2, (hi - lo) / 2))
    xs, ws = np.concatenate(xs), np.concatenate(ws)
    D2 = bspline_deriv(xs, tau, M, 2)
    return D2.T @ (ws[:, None] * D2)


class SmoothingSpline:
    """Smoothing spline (5.70)-(5.83): minimise sum (Y_i - r(x_i))^2 + lam int r''(t)^2 dt, where t is
    x rescaled to [0, 1] (so lam is free of the units of x). Knots at the distinct x values."""

    def __init__(self, x, y, lam=None, df=None, weights=None):
        self.x, self.y = np.asarray(x, float), np.asarray(y, float)
        self.a, self.b = self.x.min(), self.x.max()
        t = np.round(self.t(self.x), 10)     # values equal up to rounding share one knot
        u = np.unique(t)
        self.tau = augmented_knots(u[1:-1], 0.0, 1.0)
        self.B = bspline_basis(t, self.tau)
        self.Om = omega_matrix(self.tau)
        self.w = np.ones(len(y)) if weights is None else np.asarray(weights, float)
        self.BtB = self.B.T @ (self.w[:, None] * self.B)
        # a negligible ridge keeps A invertible when lam is tiny and a knot interval holds no data
        self.ridge = 1e-9 * np.mean(np.diag(self.BtB)) * np.eye(len(self.BtB))
        if df is not None:
            lam = spline_df_to_lam(self, df)
        self.set_lam(lam)

    def t(self, x):
        return (np.asarray(x, float) - self.a) / (self.b - self.a)

    def set_lam(self, lam):
        self.lam = float(lam)
        A = self.BtB + self.lam * self.Om + self.ridge
        self.Ainv_Bt = np.linalg.solve(A, self.B.T * self.w[None, :])
        self.beta = self.Ainv_Bt @ self.y
        self.L = self.B @ self.Ainv_Bt          # (5.82)
        self.df = float(np.trace(self.L))
        return self

    def __call__(self, xg):
        return bspline_basis(np.clip(self.t(xg), 0, 1), self.tau) @ self.beta

    def weights_at(self, xg):
        return bspline_basis(np.clip(self.t(xg), 0, 1), self.tau) @ self.Ainv_Bt


def spline_df_to_lam(ss, df):
    """lam with tr(L) = df (tr L decreases from the number of distinct x's to 2 as lam grows)."""
    def tr(loglam):
        A = ss.BtB + math.exp(loglam) * ss.Om + ss.ridge
        return float(np.trace(ss.B @ np.linalg.solve(A, ss.B.T * ss.w[None, :]))) - df
    # beyond lam_max the matrix is numerically lam * Omega, whose null space (the lines) makes it singular
    hi = math.log(1e9 * np.max(np.diag(ss.BtB)) / np.max(np.abs(ss.Om)))
    return math.exp(optimize.brentq(tr, hi - 60.0, hi, xtol=1e-10))


def spline_select(x, y, lams, score="cv"):
    """CV and GCV over a grid of lam values for the smoothing spline."""
    ss = SmoothingSpline(x, y, lam=lams[0])
    out = {"lam": np.asarray(lams, float), "cv": [], "gcv": [], "df": []}
    for lam in lams:
        ss.set_lam(lam)
        out["cv"].append(loo_cv(ss.L, ss.y)); out["gcv"].append(gcv(ss.L, ss.y)); out["df"].append(ss.df)
    for k in ("cv", "gcv", "df"):
        out[k] = np.array(out[k])
    out["best"] = int(np.argmin(out[score]))
    ss.set_lam(lams[out["best"]])
    out["fit"] = ss
    return out


def truncated_power_basis(x, knots):
    """h_1..h_{k+4} of Theorem 5.74: 1, x, x^2, x^3, (x - xi_j)_+^3."""
    x = np.asarray(x, float)
    cols = [np.ones_like(x), x, x ** 2, x ** 3] + [np.clip(x - k, 0, None) ** 3 for k in knots]
    return np.stack(cols, 1)


def regression_spline(x, y, nknots, xg=None):
    """Cubic regression spline with nknots equally spaced interior knots: L = B (B^T B)^{-1} B^T."""
    a, b = np.min(x), np.max(x)
    inner = np.linspace(a, b, nknots + 2)[1:-1]
    tau = augmented_knots(inner, a, b)
    B = bspline_basis(x, tau)
    H = B @ np.linalg.pinv(B.T @ B) @ B.T
    if xg is None:
        return H
    Bg = bspline_basis(np.clip(xg, a, b), tau)
    return H, Bg @ np.linalg.pinv(B.T @ B) @ B.T


def silverman_kernel(t):
    """Silverman's (1984) equivalent kernel of the cubic smoothing spline."""
    a = np.abs(t) / math.sqrt(2)
    return 0.5 * np.exp(-a) * np.sin(a + math.pi / 4)


# =========================================================================== Section 5.6: variance
def sigma2_linear(L, y):
    """(5.86): ||Y - LY||^2 / (n - 2 nu + nu~), nu = tr L, nu~ = tr(L^T L)."""
    r = y - L @ y
    n = len(y)
    return float(r @ r / (n - 2 * np.trace(L) + np.sum(L * L)))


def sigma2_rice(x, y):
    """Rice's first-difference estimator (5.89), with the x's sorted."""
    o = np.argsort(x, kind="stable")
    d = np.diff(np.asarray(y)[o])
    return float(d @ d / (2 * (len(y) - 1)))


def sigma2_gasser(x, y):
    """Gasser, Sroka and Jennen-Steinmetz (1986) pseudo-residual estimator (5.91)."""
    o = np.argsort(x, kind="stable")
    xs, ys = np.asarray(x)[o], np.asarray(y)[o]
    a = (xs[2:] - xs[1:-1]) / (xs[2:] - xs[:-2])
    b = (xs[1:-1] - xs[:-2]) / (xs[2:] - xs[:-2])
    c2 = 1 / (a * a + b * b + 1)
    delta = a * ys[:-2] + b * ys[2:] - ys[1:-1]
    return float(np.sum(c2 * delta ** 2) / (len(ys) - 2))


LOG_CHI2_MEAN = float(-np.euler_gamma - math.log(2))     # E log(eps^2) = psi(1/2) + log 2 = -1.2704 for N(0, 1)


def variance_function(x, y, fit_resid, hs, kind="tricube", correct=True):
    """Steps 2-3 of the variance function estimation (5.95): local linear regression of
    Z_i = log(residual_i^2) on x with the bandwidth chosen by cross-validation. Returns (q at x, h, sel).
    The regression estimates E Z = log sigma^2(x) + E log eps^2, so with correct=True the constant
    E log eps^2 = -1.2704 of Normal errors is removed (the book's recipe, correct=False, estimates
    sigma^2(x) times exp(-1.2704) = 0.28)."""
    Z = np.log(fit_resid ** 2)
    sel = select_h(x, Z, hs, "ll", kind)
    L = ll_matrix(x, x, sel["hbest"], kind)
    q = L @ Z
    return (q - LOG_CHI2_MEAN if correct else q), sel["hbest"], sel


# =========================================================================== Section 5.7: bands
def tube_kappa0(Lg, xg):
    """kappa_0 = int ||T'(x)|| dx (5.101), T(x) = l(x)/||l(x)||, from the weights Lg on the grid xg
    (central differences, trapezoid rule)."""
    T = Lg / np.linalg.norm(Lg, axis=1, keepdims=True)
    dT = np.gradient(T, xg, axis=0)
    return float(np.trapz(np.linalg.norm(dT, axis=1), xg))


def tube_c(kappa0, alpha=0.05, m=None):
    """Solve (5.102) for c; with m (= n - tr L) use the t version of Sun and Loader (1994)."""
    if m is None:
        f = lambda c: 2 * (1 - stats.norm.cdf(c)) + kappa0 / math.pi * math.exp(-c * c / 2) - alpha
    else:
        f = lambda c: 2 * stats.t.sf(c, m) + kappa0 / math.pi * (1 + c * c / m) ** (-m / 2) - alpha
    return float(optimize.brentq(f, 0.5, 20))


def linear_band(Lg, fit, sigma, c):
    """r_n(x) +- c sigma(x) ||l(x)|| (5.99)/(5.104); sigma may be a scalar or a function on the grid."""
    se = np.asarray(sigma) * np.linalg.norm(Lg, axis=1)
    return fit - c * se, fit + c * se, se


# =========================================================================== Section 5.10: local likelihood
def _expit(z):
    return 1 / (1 + np.exp(-z))


def local_logistic(x, y, grid, h, kind="tricube", iters=100, tol=1e-10, return_details=False):
    """Local linear logistic regression (5.115): at each x maximise the kernel-weighted
    log-likelihood l_x(a) by Newton-Raphson. Returns r_n(grid) (and a0, a1, m(x) of (5.118))."""
    grid = np.atleast_1d(np.asarray(grid, float))
    r = np.full(len(grid), np.nan)
    A = np.full((len(grid), 2), np.nan)
    mm = np.full(len(grid), np.nan)
    a = np.zeros(2)
    for g, x0 in enumerate(grid):
        d = x - x0
        w = kernel(d / h, kind)
        if np.sum(w > 0) < 3:
            continue
        X = np.stack([np.ones_like(d), d], 1)
        a = np.array([math.log((np.sum(w * y) + .5) / (np.sum(w * (1 - y)) + .5)), 0.0])
        for _ in range(iters):
            p = _expit(X @ a)
            grad = X.T @ (w * (y - p))
            Hs = X.T @ ((w * p * (1 - p))[:, None] * X) + 1e-10 * np.eye(2)
            step = np.linalg.solve(Hs, grad)
            a = a + step
            if np.max(np.abs(step)) < tol:
                break
        if not np.all(np.isfinite(a)):
            continue
        A[g] = a
        r[g] = _expit(a[0])
        p = _expit(X @ a)
        mm[g] = kernel(0.0, kind) * np.linalg.solve(X.T @ ((w * p * (1 - p))[:, None] * X), [1.0, 0.0])[0]
    if return_details:
        return r, A, mm
    return r


def _loglik(y, xi):
    return y * xi - np.logaddexp(0, xi)


def local_logistic_cv(x, y, h, kind="tricube"):
    """Leave-one-out log-likelihood (5.116) by refitting n times, and the approximation (5.117),
    which is l_x-free: sum_i l(Y_i, xi_hat(x_i)) - sum_i m(x_i) (Y_i - p_i)^2."""
    n = len(y)
    exact = 0.0
    for i in range(n):
        keep = np.arange(n) != i
        r = local_logistic(x[keep], y[keep], [x[i]], h, kind)[0]
        if not np.isfinite(r):
            return -np.inf, -np.inf
        r = min(max(r, 1e-12), 1 - 1e-12)
        exact += _loglik(y[i], math.log(r / (1 - r)))
    r, A, mm = local_logistic(x, y, x, h, kind, return_details=True)
    if not np.all(np.isfinite(r)):
        return exact, -np.inf
    approx = np.sum(_loglik(y, A[:, 0])) - np.sum(mm * (y - r) ** 2)
    return exact, float(approx)


# =========================================================================== Section 5.12: multiple regression
def loclin_nd(X, Y, Xg, h, kind="tricube"):
    """Multivariate local linear regression (5.122)-(5.124) with weights K(||x_i - x||/h).
    Returns the weight matrix (rows = rows of Xg)."""
    X, Xg = np.atleast_2d(X), np.atleast_2d(Xg)
    n, d = X.shape
    out = np.full((len(Xg), n), np.nan)
    for g, x0 in enumerate(Xg):
        D = X - x0
        w = kernel(np.linalg.norm(D, axis=1) / h, kind)
        Xx = np.hstack([np.ones((n, 1)), D])
        M = Xx.T @ (w[:, None] * Xx)
        if np.linalg.cond(M) > 1e12:
            continue
        out[g] = np.linalg.solve(M, Xx.T * w[None, :])[0]
    return out


def backfit(X, Y, hs_grid=None, h=None, iters=50, tol=1e-8, kind="tricube"):
    """The backfitting algorithm (Section 5.12) with local linear smoothers. If h is None, each
    bandwidth is re-chosen by cross-validation at every pass, as in Example 5.129. Returns alpha,
    the fitted components at the data (n x d) and the bandwidths."""
    n, d = X.shape
    alpha = Y.mean()
    R = np.zeros((n, d))
    hh = np.array(h if h is not None else [np.ptp(X[:, j]) / 4 for j in range(d)], float)
    for it in range(iters):
        old = R.copy()
        for j in range(d):
            part = Y - alpha - R.sum(1) + R[:, j]
            if h is None:
                grid = hs_grid[j]
                sel = select_h(X[:, j], part, grid, "ll", kind)
                hh[j] = sel["hbest"]
            Lj = ll_matrix(X[:, j], X[:, j], hh[j], kind)
            fj = Lj @ part
            R[:, j] = fj - fj.mean()
        if np.max(np.abs(R - old)) < tol:
            break
    return alpha, R, hh, it + 1


def _ss_gcv_fit(z, r, spans=np.linspace(0.25, 1.5, 11)):
    """The smoother S of projection pursuit: local linear regression (tricube) with bandwidth
    h = span * range(z), the span chosen by GCV (5.36); spans of at least a quarter of the range keep
    a ridge function from interpolating a small data set. Returns the fitted values, a predictor and h."""
    best = None
    for sp in spans:
        h = sp * np.ptp(z)
        L = ll_matrix(z, z, h)
        if not np.all(np.isfinite(L)):
            continue
        g = gcv(L, r)
        if best is None or g < best[0]:
            best = (g, h, L)
    _, h, L = best
    return L @ r, (lambda zn: ll_matrix(z, np.asarray(zn, float), h) @ r), h


def ppr(X, Y, M=3, restarts=4, seed=0):
    """Projection pursuit regression (Friedman and Stuetzle 1981) as in Section 5.12: forward
    stagewise terms r_m(alpha_m^T x), each direction found by maximising I(alpha) over the unit
    sphere (random restarts + Nelder-Mead), each ridge function a local linear fit with its
    bandwidth chosen by GCV (the book used R's ppr with smoothing splines). X should be
    standardised; Y is centred inside. Returns mu, the directions, the fitted terms at the data and
    the residual sums of squares after each term."""
    rng = np.random.default_rng(seed)
    n, d = X.shape
    mu = Y.mean()
    e = Y - mu
    dirs, terms, rss = [], [], [float(e @ e)]

    def unit(th):
        a = np.array(th[:d])
        return a / np.linalg.norm(a)

    def crit(th):
        a = unit(th)
        f, _, _ = _ss_gcv_fit(X @ a, e)
        return np.sum((e - f) ** 2)

    for m in range(M):
        best = None
        starts = [np.eye(d)[j] for j in range(d)] + [rng.normal(size=d) for _ in range(restarts)]
        for s in starts:
            res = optimize.minimize(crit, s, method="Nelder-Mead", options={"xatol": 1e-4, "fatol": 1e-8, "maxiter": 300})
            if best is None or res.fun < best.fun:
                best = res
        a = unit(best.x)
        if a[np.argmax(np.abs(a))] < 0:
            a = -a
        f, _, _ = _ss_gcv_fit(X @ a, e)
        dirs.append(a); terms.append(f)
        e = e - f
        rss.append(float(e @ e))
    return mu, np.array(dirs), np.array(terms).T, np.array(rss)


def tree_fit(X, Y, seed=0, folds=10):
    """CART regression tree: grow a large tree, prune by cost complexity (5.132), alpha by 10-fold CV.
    Uses scikit-learn's DecisionTreeRegressor (CART, Breiman et al. 1984)."""
    from sklearn.model_selection import KFold
    from sklearn.tree import DecisionTreeRegressor
    big = DecisionTreeRegressor(min_samples_leaf=2, random_state=seed).fit(X, Y)
    path = big.cost_complexity_pruning_path(X, Y)
    alphas = path.ccp_alphas[:-1]
    kf = KFold(folds, shuffle=True, random_state=seed)
    err = np.zeros(len(alphas))
    for tr, te in kf.split(X):
        for k, al in enumerate(alphas):
            t = DecisionTreeRegressor(min_samples_leaf=2, random_state=seed, ccp_alpha=al).fit(X[tr], Y[tr])
            err[k] += np.sum((Y[te] - t.predict(X[te])) ** 2)
    k = int(np.argmin(err))
    tree = DecisionTreeRegressor(min_samples_leaf=2, random_state=seed, ccp_alpha=alphas[k]).fit(X, Y)
    return tree, alphas, err / len(Y)


def _hinge_pair(xj, t):
    return np.clip(xj - t, 0, None), np.clip(t - xj, 0, None)


def mars(X, Y, max_terms=15, degree=2, penalty=3.0):
    """A compact MARS (Friedman 1991; Hastie et al. 2001, Section 9.4): forward pass adding reflected
    pairs (x_j - t)_+, (t - x_j)_+ (knots at the data, products with existing terms up to `degree`),
    then backward deletion by GCV with cost M + penalty (M - 1)/2. Returns predict(X) and the terms."""
    n, d = X.shape
    basis = [np.ones(n)]
    desc = [()]
    deg = [0]

    def rss_of(cols):
        Bm = np.stack(cols, 1)
        coef = np.linalg.lstsq(Bm, Y, rcond=None)[0]
        r = Y - Bm @ coef
        return r @ r

    while len(basis) < max_terms:
        best = None
        for p_idx, parent in enumerate(basis):
            if deg[p_idx] >= degree:
                continue
            used = {u[0] for u in desc[p_idx]}
            for j in range(d):
                if j in used:
                    continue
                for t in np.unique(X[:, j])[1:-1]:
                    h1, h2 = _hinge_pair(X[:, j], t)
                    cand = basis + [parent * h1, parent * h2]
                    s = rss_of(cand)
                    if best is None or s < best[0]:
                        best = (s, p_idx, j, t)
        if best is None:
            break
        _, p_idx, j, t = best
        h1, h2 = _hinge_pair(X[:, j], t)
        basis += [basis[p_idx] * h1, basis[p_idx] * h2]
        desc += [desc[p_idx] + ((j, t, +1),), desc[p_idx] + ((j, t, -1),)]
        deg += [deg[p_idx] + 1, deg[p_idx] + 1]

    def gcv_of(idx):
        cols = [basis[i] for i in idx]
        s = rss_of(cols)
        Mc = len(idx) + penalty * (len(idx) - 1) / 2
        return s / n / (1 - Mc / n) ** 2 if Mc < n else np.inf

    cur = list(range(len(basis)))
    best_set, best_g = list(cur), gcv_of(cur)
    while len(cur) > 1:
        trials = [(gcv_of([i for i in cur if i != k]), k) for k in cur if k != 0]
        g, k = min(trials)
        cur = [i for i in cur if i != k]
        if g < best_g:
            best_g, best_set = g, list(cur)
    cols = np.stack([basis[i] for i in best_set], 1)
    coef = np.linalg.lstsq(cols, Y, rcond=None)[0]
    terms = [desc[i] for i in best_set]

    def predict(Xn):
        out = np.zeros(len(Xn))
        for c, tm in zip(coef, terms):
            v = np.ones(len(Xn))
            for j, t, s in tm:
                v = v * np.clip(s * (Xn[:, j] - t), 0, None)
            out += c * v
        return out
    return predict, terms, coef, best_g


# =========================================================================== Section 5.13: other issues
def plugin_h(x, y, kind="tricube"):
    """The crude plug-in bandwidth (5.138): a global quartic pilot, f uniform on the range of x."""
    X = np.vander(x, 5, increasing=True)
    beta = np.linalg.lstsq(X, y, rcond=None)[0]
    s2 = np.mean((y - X @ beta) ** 2)
    r2 = 2 * beta[2] + 6 * beta[3] * x + 12 * beta[4] * x ** 2
    u = np.linspace(-1, 1, 20001)
    Ku = kernel(u, kind)
    C = np.trapz(Ku ** 2, u) / np.trapz(u * u * Ku, u) ** 2
    return float((C * s2 * np.ptp(x) / np.sum(r2 ** 2)) ** 0.2), C


def linearity_test(x, y, h, kind="tricube"):
    """Loader's F approximation for H0: r linear. T = (||LY - HY||^2 / lam) / sigma2 with
    lam = tr((L - H)^T (L - H)) and sigma2 from (5.86); reference F(lam, n - 2 nu1 + nu2)."""
    n = len(y)
    L = ll_matrix(x, x, h, kind)
    X = np.stack([np.ones(n), x], 1)
    H = X @ np.linalg.solve(X.T @ X, X.T)
    D = L - H
    lam = np.sum(D * D)
    s2 = sigma2_linear(L, y)
    T = np.sum((D @ y) ** 2) / lam / s2
    df2 = n - 2 * np.trace(L) + np.sum(L * L)
    return T, lam, df2, float(stats.f.sf(T, lam, df2))


def robust_local(x, y, grid, h, rho="huber", c=1.345, alpha=0.5, kind="tricube", iters=200):
    """Local linear M-estimation (5.149) by iteratively reweighted least squares: Huber's psi (5.150)
    or the quantile check loss (5.151) (alpha-quantile)."""
    grid = np.atleast_1d(grid)
    out = np.full(len(grid), np.nan)
    for g, x0 in enumerate(grid):
        d = x - x0
        w = kernel(d / h, kind)
        keep = w > 0
        if keep.sum() < 3:
            continue
        X = np.stack([np.ones(keep.sum()), d[keep]], 1)
        yy, ww = y[keep], w[keep]
        a = np.linalg.lstsq(X * np.sqrt(ww)[:, None], yy * np.sqrt(ww), rcond=None)[0]
        s = 1.4826 * np.median(np.abs(yy - X @ a)) + 1e-12
        for _ in range(iters):
            r = yy - X @ a
            if rho == "huber":
                u = np.abs(r) / s
                v = np.where(u <= c, 1.0, c / np.maximum(u, 1e-12))
            else:   # quantile: |t| + (2 alpha - 1) t, IRLS weight = rho(t)/t^2
                ar = np.maximum(np.abs(r), 1e-8 * s)
                v = np.where(r >= 0, alpha, 1 - alpha) / ar
            ws = ww * v
            new = np.linalg.solve(X.T @ (ws[:, None] * X) + 1e-12 * np.eye(2), X.T @ (ws * yy))
            if np.max(np.abs(new - a)) < 1e-9 * (1 + np.max(np.abs(a))):
                a = new
                break
            a = new
        out[g] = a[0]
    return out


# =========================================================================== chapter settings
H_CMB400 = log_grid(3, 200, 120)        # bandwidth grid for the first 400 CMB points
H_CMB = log_grid(5, 400, 120)           # ... for all 899 points
H_LIDAR = np.arange(10.0, 151.0, 1.0)
H_LOGVAR = log_grid(20, 1500, 90)
SEED_DOPPLER = 5063
SEED_BUMPS = 598
SEED_LOGIT = 5119


def doppler_data(n=1000, sigma=0.1, seed=SEED_DOPPLER):
    """Example 5.63: Y_i = r(i/n) + sigma eps_i."""
    x = np.arange(1, n + 1) / n
    rng = np.random.default_rng(seed)
    return x, doppler(x) + sigma * rng.standard_normal(n)


def cmb_full_fit():
    """Local linear fit to all 899 CMB points with the bandwidth chosen by cross-validation on the
    first 400 points (where the variance is roughly constant); returns x, y, se, h, L."""
    x4, y4, _ = cmb(400)
    h = select_h(x4, y4, H_CMB400, "ll")["hbest"]
    x, y, se = cmb()
    return x, y, se, h, ll_matrix(x, x, h)


def _style(ax, xl=None, yl=None):
    if xl:
        ax.set_xlabel(xl)
    if yl:
        ax.set_ylabel(yl)


# =========================================================================== figures
def fig_5_1(out=None):
    """CMB data: all 899 points and the first 400."""
    x, y, _ = cmb()
    fig, ax = common.panel_grid(2, 1, size=(6.2, 2.5))
    ax[0, 0].plot(x, y, ".", ms=2.5, color=BLUE)
    ax[1, 0].plot(x[:400], y[:400], ".", ms=3, color=BLUE)
    _style(ax[0, 0], "multipole", "power"); _style(ax[1, 0], "multipole", "power")
    fig.tight_layout()
    savefig(fig, CH, "fig5-1", out)


def fig_5_2(out=None):
    """Four Nadaraya-Watson fits to the first 400 CMB points, h = 1, 10, 50, 200 (tricube)."""
    x, y, _ = cmb(400)
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    for a, h in zip(ax.ravel(), [1, 10, 50, 200]):
        L = nw_matrix(x, x, h)
        a.plot(x, y, ".", ms=2, color=GREY, alpha=.6)
        a.plot(x, L @ y, color=RED, lw=1.5)
        a.set_title("h = %g (%.1f df)" % (h, np.trace(L)))
        _style(a, "multipole", "power")
    fig.tight_layout()
    savefig(fig, CH, "fig5-2", out)


def cmb_kernel_cv():
    x, y, _ = cmb()
    sel = select_h(x, y, H_CMB, "nw")
    df, cv = sel["df"], sel["cv"]
    # the local minimum in the range of the book's plot (20 to 35 degrees of freedom)
    win = np.where((df > 20) & (df < 35))[0]
    loc = win[np.argmin(cv[win])]
    return x, y, sel, loc


def fig_5_3(out=None):
    """CV score against effective degrees of freedom (all CMB data, kernel), and the fit."""
    x, y, sel, loc = cmb_kernel_cv()
    fig, ax = common.panel_grid(2, 1, size=(6.2, 2.6))
    a = ax[0, 0]
    ok = np.isfinite(sel["cv"]) & (sel["df"] < 60)
    a.plot(sel["df"][ok], sel["cv"][ok] / 1e6, color=BLUE)
    a.axvline(sel["df"][loc], color=RED, ls="--", lw=1)
    a.axvline(sel["df"][sel["best"]], color=GREY, ls=":", lw=1)
    a.text(sel["df"][loc] + 0.5, a.get_ylim()[1] * 0.995, "local min, %.1f df" % sel["df"][loc], va="top", fontsize=8, color=RED)
    a.text(sel["df"][sel["best"]] + 0.5, a.get_ylim()[1] * 0.97, "global min, %.1f df" % sel["df"][sel["best"]], va="top", fontsize=8, color=GREY)
    _style(a, r"effective degrees of freedom $\nu$", r"CV score ($\times 10^6$)")
    L = nw_matrix(x, x, sel["h"][loc])
    b = ax[1, 0]
    b.plot(x, L @ y, color=RED)
    b.set_ylim(0, 6500)
    _style(b, "multipole", "power")
    b.set_title("kernel fit, h = %.1f" % sel["h"][loc])
    fig.tight_layout()
    savefig(fig, CH, "fig5-3", out)


def lidar_analysis():
    """Example 5.59 / 5.110: the fit, residuals, sigma(x) and the 95 percent band."""
    x, y = lidar()
    sel = select_h(x, y, H_LIDAR, "ll")
    h = sel["hbest"]
    L = ll_matrix(x, x, h)
    res = y - L @ y
    q, hv, selv = variance_function(x, y, res, H_LOGVAR)
    xg = np.linspace(x.min(), x.max(), 600)
    Lg = ll_matrix(x, xg, h)
    sig_g = np.exp(0.5 * (ll_matrix(x, xg, hv) @ np.log(res ** 2)))
    k0 = tube_kappa0(Lg, xg)
    c = tube_c(k0)
    s_exact = np.sqrt((Lg ** 2) @ np.exp(q))            # (5.103)
    fit = Lg @ y
    return dict(x=x, y=y, h=h, df=np.trace(L), L=L, res=res, q=q, hv=hv, xg=xg, Lg=Lg, sig_g=sig_g,
                kappa0=k0, c=c, fit=fit, lo=fit - c * s_exact, up=fit + c * s_exact, sel=sel, selv=selv)


def fig_5_4(out=None):
    """LIDAR: local linear fit, residuals, sigma(x), 95 percent bands."""
    A = lidar_analysis()
    x, y = A["x"], A["y"]
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    ax[0, 0].plot(x, y, ".", ms=3, color=GREY); ax[0, 0].plot(A["xg"], A["fit"], color=RED)
    ax[0, 0].set_title("h = %.0f (%.1f df)" % (A["h"], A["df"]))
    ax[0, 1].plot(x, A["res"], ".", ms=3, color=BLUE); ax[0, 1].axhline(0, color=GREY, lw=.8)
    ax[1, 0].plot(A["xg"], A["sig_g"], color=RED); ax[1, 0].set_title(r"$\hat\sigma(x)$, h = %.0f" % A["hv"])
    ax[1, 1].plot(x, y, ".", ms=3, color=GREY); ax[1, 1].plot(A["xg"], A["fit"], color=RED)
    ax[1, 1].plot(A["xg"], A["lo"], color=BLUE, lw=1); ax[1, 1].plot(A["xg"], A["up"], color=BLUE, lw=1)
    ax[1, 1].set_title(r"95%% band, $\kappa_0$ = %.1f, c = %.2f" % (A["kappa0"], A["c"]))
    for a, yl in zip(ax.ravel(), ["log ratio", "residuals", "sigma(x)", "log ratio"]):
        _style(a, "range", yl)
    fig.tight_layout()
    savefig(fig, CH, "fig5-4", out)


def fig_5_5(out=None):
    """CMB, first 400 points: local constant (p = 0) and local linear (p = 1); left boundary."""
    x, y, _ = cmb(400)
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    for j, (m, lab) in enumerate([("nw", "p = 0"), ("ll", "p = 1")]):
        h = select_h(x, y, H_CMB400, m)["hbest"]
        Lf = smoother_matrix(x, h, m)
        f = Lf @ y
        ax[0, j].plot(x, y, ".", ms=2, color=GREY, alpha=.6); ax[0, j].plot(x, f, color=RED)
        ax[0, j].set_title("%s, h = %.1f" % (lab, h)); ax[0, j].set_ylim(-500, 6500)
        k = x <= 45
        ax[1, j].plot(x[k], y[k], "o", ms=3, color=GREY); ax[1, j].plot(x[k], f[k], color=RED, marker=".", ms=3)
        ax[1, j].set_title("%s, left boundary" % lab)
        for a in ax[:, j]:
            _style(a, "multipole", "power")
    fig.tight_layout()
    savefig(fig, CH, "fig5-5", out)


def doppler_analysis(sigma=0.1, n=1000, seed=SEED_DOPPLER):
    x, y = doppler_data(n, sigma, seed)
    sel = select_h(x, y, log_grid(0.002, 0.2, 70), "ll")
    return x, y, sel


def fig_5_6(out=None):
    """The Doppler function: function, data, CV against df, local linear fit."""
    x, y, sel = doppler_analysis()
    h = sel["hbest"]
    L = ll_matrix(x, x, h)
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    xx = np.linspace(0, 1, 2000)
    ax[0, 0].plot(xx, doppler(xx), color=BLUE); ax[0, 0].set_title("Doppler function")
    ax[0, 1].plot(x, y, ".", ms=2, color=GREY); ax[0, 1].set_title("n = 1000, sigma = 0.1")
    ok = np.isfinite(sel["cv"])
    ax[1, 0].plot(sel["df"][ok], sel["cv"][ok], color=BLUE); ax[1, 0].set_xlim(0, 400)
    ax[1, 0].axvline(sel["df"][sel["best"]], color=RED, ls="--", lw=1)
    ax[1, 0].set_title("CV minimum: %.0f df, h = %.4f" % (sel["df"][sel["best"]], h))
    _style(ax[1, 0], "effective degrees of freedom", "CV score")
    ax[1, 1].plot(x, L @ y, color=RED, lw=1); ax[1, 1].set_title("local linear fit")
    fig.tight_layout()
    savefig(fig, CH, "fig5-6", out)


def fig_5_7(out=None):
    """Cubic B-spline basis with nine equally spaced interior knots on (0, 1)."""
    tau = augmented_knots(np.linspace(0, 1, 11)[1:-1], 0, 1)
    xx = np.linspace(0, 1, 801)
    B = bspline_basis(xx, tau)
    fig, ax = common.panel_grid(1, 1, size=(6.2, 2.6))
    cols = [RED, BLUE, GREEN, MAGENTA, ORANGE, PURPLE, GREY]
    for j in range(B.shape[1]):
        ax[0, 0].plot(xx, B[:, j], color=cols[j % len(cols)], lw=1.3)
    for k in tau[4:-4]:
        ax[0, 0].axvline(k, color=GREY, lw=.5, ls=":")
    ax[0, 0].set_ylim(0, 1.2)
    fig.tight_layout()
    savefig(fig, CH, "fig5-7", out)


def cmb_spline():
    x, y, _ = cmb()
    return x, y, spline_select(x, y, log_grid(1e-11, 1e-1, 81))


def fig_5_8(out=None):
    """Smoothing spline for the CMB data, lambda by cross-validation."""
    x, y, sel = cmb_spline()
    ss = sel["fit"]
    fig, ax = common.panel_grid(1, 1, size=(6.2, 2.8))
    xx = np.linspace(x.min(), x.max(), 1500)
    ax[0, 0].plot(xx, ss(xx), color=RED)
    ax[0, 0].set_title("smoothing spline, %.1f effective df" % ss.df)
    ax[0, 0].set_ylim(0, 6500)
    _style(ax[0, 0], "multipole", "power")
    fig.tight_layout()
    savefig(fig, CH, "fig5-8", out)


def cmb_variance():
    x, y, se, h, L = cmb_full_fit()
    res = y - L @ y
    q, hv, selv = variance_function(x, y, res, H_LOGVAR)
    return dict(x=x, y=y, se=se, h=h, L=L, res=res, q=q, hv=hv)


def fig_5_9(out=None):
    """Log squared residuals, log sigma2_hat(x) and the log of the WMAP team's sigma^2(x)."""
    V = cmb_variance()
    fig, ax = common.panel_grid(1, 1, size=(6.2, 3.0))
    a = ax[0, 0]
    a.plot(V["x"], np.log(V["res"] ** 2), ".", ms=2, color=GREY, alpha=.7)
    a.plot(V["x"], V["q"], color=RED, label=r"estimated $\log\hat\sigma^2(x)$, h = %.0f" % V["hv"])
    a.plot(V["x"], V["q"] + LOG_CHI2_MEAN, color=ORANGE, ls=":", lw=1.6, label="without the correction 1.27")
    a.plot(V["x"], np.log(V["se"] ** 2), color=BLUE, ls="--", label=r"$\log\sigma^2(x)$ from WMAP")
    a.legend(loc="lower right")
    _style(a, "multipole", r"$\log\sigma^2(x)$")
    fig.tight_layout()
    savefig(fig, CH, "fig5-9", out)


def bumps_analysis(seed=SEED_BUMPS):
    x = np.arange(1, 101) / 10
    rng = np.random.default_rng(seed)
    y = bumps4(x) + 0.2 * rng.standard_normal(100)
    xx = np.linspace(0.1, 10, 1000)
    Lg = ll_matrix(x, xx, 0.27)
    return x, y, xx, Lg


def fig_5_10(out=None):
    """r, r_n (local linear, h = 0.27), r_bar_n = E r_n and r - r_bar_n for the four-bump function."""
    x, y, xx, Lg = bumps_analysis()
    rbar = Lg @ bumps4(x)
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    ax[0, 0].plot(xx, bumps4(xx), color=BLUE); ax[0, 0].set_title("r(x)")
    ax[0, 1].plot(x, y, ".", ms=3, color=GREY); ax[0, 1].plot(xx, Lg @ y, color=RED, ls="--"); ax[0, 1].set_title(r"$\hat r_n(x)$")
    ax[1, 0].plot(xx, rbar, color=RED, ls="--"); ax[1, 0].set_title(r"$\bar r_n(x)=E\,\hat r_n(x)$")
    ax[1, 1].plot(xx, bumps4(xx) - rbar, color=PURPLE); ax[1, 1].set_title(r"$r(x)-\bar r_n(x)$")
    for a in ax[0]:
        a.set_ylim(-0.5, 8.5)
    ax[1, 0].set_ylim(-0.5, 8.5)
    fig.tight_layout()
    savefig(fig, CH, "fig5-10", out)


def cmb_bands():
    """Example 5.105: local linear fit to all CMB data, 95 percent bands with constant variance and
    with the variance function."""
    V = cmb_variance()
    x, y, h, L = V["x"], V["y"], V["h"], V["L"]
    xg = np.linspace(x.min(), x.max(), 1500)
    Lg = ll_matrix(x, xg, h)
    k0 = tube_kappa0(Lg, xg)
    c = tube_c(k0)
    fit = Lg @ y
    s_const = math.sqrt(sigma2_linear(L, y))
    sig2_i = np.exp(V["q"])
    s_var = np.sqrt((Lg ** 2) @ sig2_i)
    return dict(V, xg=xg, Lg=Lg, kappa0=k0, c=c, fit=fit, s_const=s_const,
                lo_c=fit - c * s_const * np.linalg.norm(Lg, axis=1), up_c=fit + c * s_const * np.linalg.norm(Lg, axis=1),
                lo_v=fit - c * s_var, up_v=fit + c * s_var)


def fig_5_11(out=None):
    """CMB local linear fit with simultaneous 95 percent bands: constant and non-constant variance."""
    B = cmb_bands()
    fig, ax = common.panel_grid(2, 1, size=(6.2, 2.6))
    for a, (lo, up, t) in zip(ax[:, 0], [(B["lo_c"], B["up_c"], "constant variance"), (B["lo_v"], B["up_v"], r"variance function $\hat\sigma^2(x)$")]):
        a.fill_between(B["xg"], lo, up, color=BLUE, alpha=.25, lw=0)
        a.plot(B["xg"], B["fit"], color=RED)
        a.set_ylim(-3000, 14000)
        a.set_title(r"%s; h = %.1f, $\kappa_0$ = %.1f, c = %.2f" % (t, B["h"], B["kappa0"], B["c"]), fontsize=9.5)
        _style(a, "multipole", "power")
    fig.tight_layout()
    savefig(fig, CH, "fig5-11", out)


def logit_sim(n=300, seed=SEED_LOGIT):
    rng = np.random.default_rng(seed)
    x = np.sort(rng.uniform(-3, 3, n))
    r = 1 / (1 + np.exp(-3 * np.sin(x)))
    y = (rng.uniform(size=n) < r).astype(float)
    return x, y


def local_logistic_select(x, y, hs):
    out = []
    for h in hs:
        ex, _ = local_logistic_cv(x, y, h)
        out.append(ex)
    out = np.array(out)
    return hs[int(np.argmax(out))], out


def fig_5_12(out=None):
    """Local linear logistic regression against local linear regression on simulated Bernoulli data."""
    x, y = logit_sim()
    hs = log_grid(0.4, 4, 18)
    h1, _ = local_logistic_select(x, y, hs)
    h2 = select_h(x, y, hs, "ll")["hbest"]
    xx = np.linspace(-3, 3, 400)
    fig, ax = common.panel_grid(1, 1, size=(6.2, 3.0))
    a = ax[0, 0]
    a.plot(x, y, "|", ms=6, color=GREY)
    a.plot(xx, 1 / (1 + np.exp(-3 * np.sin(xx))), color="k", label="true r(x)")
    a.plot(xx, local_logistic(x, y, xx, h1), color=RED, ls="--", label="local logistic, h = %.2f" % h1)
    a.plot(xx, ll_matrix(x, xx, h2) @ y, color=BLUE, ls=":", lw=2, label="local linear, h = %.2f" % h2)
    a.legend(loc="center left", fontsize=8)
    fig.tight_layout()
    savefig(fig, CH, "fig5-12", out)


def bpd_analysis():
    x, y = bpd()
    X = np.stack([np.ones_like(x), x / 1000], 1)
    beta = logistic_irls(X, y)
    hs = log_grid(150, 1500, 18)
    h1, cv1 = local_logistic_select(x, y, hs)
    h2 = select_h(x, y, hs, "ll")["hbest"]
    return x, y, beta, h1, h2


def fig_5_13(out=None):
    """BPD data: logistic regression, local likelihood and local linear regression."""
    x, y, beta, h1, h2 = bpd_analysis()
    xx = np.linspace(x.min(), x.max(), 400)
    fig, ax = common.panel_grid(1, 1, size=(6.2, 3.0))
    a = ax[0, 0]
    jit = np.random.default_rng(1).uniform(-0.02, 0.02, len(y))
    a.plot(x, y + jit, "|", ms=7, color=GREY)
    a.plot(xx, _expit(beta[0] + beta[1] * xx / 1000), color="k", label="logistic regression")
    a.plot(xx, local_logistic(x, y, xx, h1), color=RED, ls="--", label="local likelihood, h = %.0f" % h1)
    a.plot(xx, ll_matrix(x, xx, h2) @ y, color=BLUE, ls=":", lw=2, label="local linear, h = %.0f" % h2)
    a.legend(loc="upper right", fontsize=8)
    _style(a, "birth weight (grams)", "bronchopulmonary dysplasia")
    fig.tight_layout()
    savefig(fig, CH, "fig5-13", out)


ROCK_NAMES = ["area", "perimeter", "shape"]


def fig_5_14(out=None):
    """The rock data: log permeability against area, perimeter and shape."""
    X, Y = rock()
    fig, ax = common.panel_grid(1, 3, size=(2.5, 2.4))
    for j in range(3):
        ax[0, j].plot(X[:, j], Y, "o", ms=3, color=BLUE)
        _style(ax[0, j], ROCK_NAMES[j], "log permeability" if j == 0 else None)
    fig.tight_layout()
    savefig(fig, CH, "fig5-14", out)


def rock_additive():
    X, Y = rock()
    grids = [np.linspace(np.ptp(X[:, j]) * 0.08, np.ptp(X[:, j]) * 1.5, 40) for j in range(3)]
    alpha, R, hh, its = backfit(X, Y, grids)
    return X, Y, alpha, R, hh, its


def rock_loclin3():
    X, Y = rock()
    Z = (X - X.mean(0)) / X.std(0)
    hs = np.linspace(1.0, 6.0, 51)
    cvs = []
    for h in hs:
        L = loclin_nd(Z, Y, Z, h)
        cvs.append(loo_cv(L, Y) if np.all(np.isfinite(L)) else np.inf)
    cvs = np.array(cvs)
    h = hs[int(np.argmin(cvs))]
    L = loclin_nd(Z, Y, Z, h)
    return Z, Y, h, L, hs, cvs


def fig_5_15(out=None):
    """Additive model for the rock data by backfitting: the three components (plus Y_bar)."""
    X, Y, alpha, R, hh, its = rock_additive()
    fig, ax = common.panel_grid(1, 3, size=(2.5, 2.4))
    for j in range(3):
        o = np.argsort(X[:, j])
        part = Y - alpha - R.sum(1) + R[:, j]          # partial residuals for component j
        ax[0, j].plot(X[:, j], alpha + part, "o", ms=2.5, color=GREY, alpha=.6)
        ax[0, j].plot(X[o, j], alpha + R[o, j], color=RED)
        ax[0, j].set_title("h = %.3g" % hh[j], fontsize=9)
        _style(ax[0, j], ROCK_NAMES[j], "log permeability" if j == 0 else None)
    fig.tight_layout()
    savefig(fig, CH, "fig5-15", out)


def fig_5_16(out=None):
    """Residuals: additive model, its normal QQ plot, 3-d local linear, and the two against each other."""
    X, Y, alpha, R, hh, its = rock_additive()
    fa = alpha + R.sum(1)
    Z, _, h3, L3, _, _ = rock_loclin3()
    f3 = L3 @ Y
    ra, r3 = Y - fa, Y - f3
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    ax[0, 0].plot(fa, ra, "o", ms=3, color=BLUE); _style(ax[0, 0], "predicted values", "residuals"); ax[0, 0].set_title("additive model")
    qq = stats.probplot(ra)
    ax[0, 1].plot(qq[0][0], qq[0][1], "o", ms=3, color=BLUE); _style(ax[0, 1], "Normal quantiles", "residuals")
    ax[1, 0].plot(f3, r3, "o", ms=3, color=BLUE); _style(ax[1, 0], "predicted values", "residuals"); ax[1, 0].set_title("3-d local linear, h = %.1f" % h3)
    ax[1, 1].plot(ra, r3, "o", ms=3, color=BLUE); _style(ax[1, 1], "additive residuals", "local linear residuals")
    for a in (ax[0, 0], ax[1, 0]):
        a.axhline(0, color=GREY, lw=.8)
    fig.tight_layout()
    savefig(fig, CH, "fig5-16", out)


def rock_ppr(M=10):
    X, Y = rock()
    Z = (X - X.mean(0)) / X.std(0)
    path = os.path.join(CACHE_DIR, "ch05_rock_ppr.npz")
    if os.path.exists(path):
        D = np.load(path)
        return Z, Y, float(D["mu"]), D["dirs"], D["terms"], D["rss"]
    mu, dirs, terms, rss = ppr(Z, Y, M=M)
    os.makedirs(CACHE_DIR, exist_ok=True)
    np.savez(path, mu=mu, dirs=dirs, terms=terms, rss=rss)
    return Z, Y, mu, dirs, terms, rss


def fig_5_17(out=None):
    """Projection pursuit regression for the rock data: r_1, r_2, r_3 and RSS against M."""
    Z, Y, mu, dirs, terms, rss = rock_ppr()
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    for m, a in enumerate(ax.ravel()[:3]):
        z = Z @ dirs[m]
        o = np.argsort(z)
        a.plot(z[o], terms[o, m], color=RED)
        a.plot(z, terms[:, m] + (Y - mu - terms[:, :m + 1].sum(1)), "o", ms=2, color=GREY, alpha=.6)
        a.set_title(r"$r_%d$, $\alpha$ = (%.2f, %.2f, %.2f)" % ((m + 1,) + tuple(dirs[m])), fontsize=9)
        _style(a, r"$z_%d=\alpha_%d^T x$" % (m + 1, m + 1))
    a = ax[1, 1]
    a.plot(np.arange(len(rss)), rss, "o-", color=BLUE, ms=4)
    _style(a, "number of terms M", "residual sum of squares")
    fig.tight_layout()
    savefig(fig, CH, "fig5-17", out)


def fig_5_19(out=None):
    """Regression tree for the rock data (CART, pruned by 10-fold cross-validation)."""
    from sklearn.tree import plot_tree
    X, Y = rock()
    tree, alphas, err = tree_fit(X, Y)
    fig, ax = common.panel_grid(1, 1, size=(6.6, 3.6))
    plot_tree(tree, feature_names=ROCK_NAMES, ax=ax[0, 0], filled=False, impurity=False, rounded=True,
              fontsize=8, precision=3, node_ids=False, proportion=False, label="none")
    fig.tight_layout()
    savefig(fig, CH, "fig5-19", out)



# =========================================================================== supplementary figures
def design_bias_experiment(n=20000, h=0.06, seed=11):
    """Exact conditional bias L r - r of the kernel and local linear estimators on a non-uniform
    design (density f(x) = 0.4 + 1.2 x on [0, 1]) for r(x) = sin(2 pi x), against (5.67)-(5.68)."""
    rng = np.random.default_rng(seed)
    u = rng.uniform(size=n)
    x = np.sort((-0.4 + np.sqrt(0.16 + 2.4 * u)) / 1.2)       # inverse cdf of f
    r = lambda t: np.sin(2 * np.pi * t)
    xg = np.linspace(0, 1, 401)
    bnw = nw_matrix(x, xg, h) @ r(x) - r(xg)
    bll = ll_matrix(x, xg, h) @ r(x) - r(xg)
    uu = np.linspace(-1, 1, 20001)
    mu2 = np.trapz(uu * uu * kernel(uu), uu)
    f = 0.4 + 1.2 * xg
    r1 = 2 * np.pi * np.cos(2 * np.pi * xg)
    r2 = -(2 * np.pi) ** 2 * np.sin(2 * np.pi * xg)
    th_nw = h * h * mu2 * (0.5 * r2 + r1 * 1.2 / f)
    th_ll = h * h * mu2 * 0.5 * r2
    return xg, bnw, bll, th_nw, th_ll


def boundary_bias_rates(hs=log_grid(0.01, 0.2, 12), n=20000):
    """Bias at the boundary point x = 0 for r(x) = x + x^2 on an equispaced design."""
    x = (np.arange(n) + 0.5) / n
    r = x + x ** 2
    nw = [abs(nw_matrix(x, [0.0], h)[0] @ r - 0.0) for h in hs]
    ll = [abs(ll_matrix(x, [0.0], h)[0] @ r - 0.0) for h in hs]
    return hs, np.array(nw), np.array(ll)


def fig_5_S1(out=None):
    """Design bias (interior) and boundary bias rates of the kernel and local linear estimators."""
    xg, bnw, bll, th_nw, th_ll = design_bias_experiment()
    hs, nw, ll = boundary_bias_rates()
    fig, ax = common.panel_grid(1, 2, size=(3.4, 2.7))
    a = ax[0, 0]
    k = (xg > 0.08) & (xg < 0.92)
    a.plot(xg[k], bnw[k], color=RED, label="kernel (p = 0)")
    a.plot(xg[k], th_nw[k], color=RED, ls=":", lw=2)
    a.plot(xg[k], bll[k], color=BLUE, label="local linear (p = 1)")
    a.plot(xg[k], th_ll[k], color=BLUE, ls=":", lw=2)
    a.axhline(0, color=GREY, lw=.6)
    a.legend(fontsize=7.5, loc="lower left")
    _style(a, "x", "bias  (dotted: (5.67), (5.68))")
    a.set_title("interior: design bias", fontsize=9.5)
    b = ax[0, 1]
    b.loglog(hs, nw, "o-", color=RED, ms=3, label="kernel: slope %.2f" % np.polyfit(np.log(hs), np.log(nw), 1)[0])
    b.loglog(hs, ll, "s-", color=BLUE, ms=3, label="local linear: slope %.2f" % np.polyfit(np.log(hs), np.log(ll), 1)[0])
    b.legend(fontsize=7.5)
    _style(b, "bandwidth h", "|bias| at x = 0")
    b.set_title("boundary: O(h) against O(h$^2$)", fontsize=9.5)
    fig.tight_layout()
    savefig(fig, CH, "fig5-S1", out)


def silverman_experiment(n=500, df=12.0):
    """Rows of the smoothing-spline matrix on an equispaced design on [0, 1] against Silverman's
    approximation l_i(x) ~ K((x_i - x)/h) / (n f h) with h = (lam/(n f))^(1/4)."""
    x = (np.arange(n) + 0.5) / n
    ss = SmoothingSpline(x, np.zeros(n), df=df)
    lam_x = ss.lam * (ss.b - ss.a) ** 3            # lam in the units of x
    h = (lam_x / n) ** 0.25
    rows = [n // 2, n // 10, 3]
    return x, ss, h, rows


def fig_5_S2(out=None):
    """Silverman's equivalent kernel for the smoothing spline."""
    x, ss, h, rows = silverman_experiment()
    n = len(x)
    fig, ax = common.panel_grid(1, 1, size=(6.2, 2.8))
    a = ax[0, 0]
    for i, c in zip(rows, [RED, BLUE, GREEN]):
        a.plot(x, ss.L[i], color=c, lw=1.6, label="row at x = %.3f" % x[i])
        a.plot(x, silverman_kernel((x - x[i]) / h) / (n * h), color="k", ls=":", lw=1.2)
    a.axhline(0, color=GREY, lw=.6)
    a.legend(fontsize=8)
    a.set_title(r"smoothing spline weights (solid) and $K((x_i-x)/h)/(nh)$ (dotted); $\nu$ = %.0f, h = %.4f" % (ss.df, h), fontsize=9)
    _style(a, r"$x_i$", r"$\ell_i(x)$")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S2", out)


def band_coverage(hs=(0.03, 0.05, 0.08, 0.12, 0.18), n=200, sigma=0.3, reps=4000, alpha=0.05, seed=7):
    """Coverage of the tube-formula band (known sigma) for E r_n and for r, r(x) = sin(2 pi x) + 2x^2,
    equispaced design, local linear tricube. Returns rows (h, kappa0, c, cover_mean, cover_r)."""
    rng = np.random.default_rng(seed)
    x = (np.arange(n) + 0.5) / n
    r = lambda t: np.sin(2 * np.pi * t) + 2 * t * t
    xg = np.linspace(0, 1, 400)
    out = []
    E = rng.standard_normal((reps, n)) * sigma
    for h in hs:
        Lg = ll_matrix(x, xg, h)
        k0 = tube_kappa0(Lg, xg)
        c = tube_c(k0, alpha)
        nl = np.linalg.norm(Lg, axis=1)
        rbar = Lg @ r(x)
        noise = E @ Lg.T                                   # r_n - E r_n on the grid
        z_mean = np.max(np.abs(noise) / (sigma * nl), axis=1)
        z_true = np.max(np.abs(noise + rbar - r(xg)) / (sigma * nl), axis=1)
        out.append((h, k0, c, float(np.mean(z_mean <= c)), float(np.mean(z_true <= c))))
    return np.array(out)


def fig_5_S3(out=None):
    """Coverage of the 95 percent tube band: for E r_n (as designed) and for r (the bias problem)."""
    T = band_coverage()
    fig, ax = common.panel_grid(1, 1, size=(5.0, 2.8))
    a = ax[0, 0]
    a.plot(T[:, 0], T[:, 3], "o-", color=BLUE, label=r"covers $\bar r_n=E\,\hat r_n$ everywhere")
    a.plot(T[:, 0], T[:, 4], "s-", color=RED, label=r"covers $r$ everywhere")
    a.axhline(0.95, color=GREY, ls="--", lw=1)
    a.set_ylim(0, 1.02)
    a.legend(fontsize=8, loc="lower left")
    _style(a, "bandwidth h", "simultaneous coverage")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S3", out)


# =========================================================================== exercise figures
def regressogram_cv(x, y, ms):
    a, b = x.min(), x.max()
    return np.array([loo_cv(regressogram_matrix(x, a, b, m), y) for m in ms])


def glass_four():
    """Exercise 3: regressogram, kernel, local linear and smoothing spline fits to RI on Al, each
    tuned by leave-one-out cross-validation, sigma from (5.86), 95 percent bands."""
    x, y = glass()
    o = np.argsort(x, kind="stable")
    x, y = x[o], y[o]
    xg = np.linspace(x.min(), x.max(), 500)
    res = {}
    ms = np.arange(2, 41)
    cvm = regressogram_cv(x, y, ms)
    m = int(ms[np.argmin(cvm)])
    L = regressogram_matrix(x, x.min(), x.max(), m)
    jg = np.clip(np.floor((xg - x.min()) / np.ptp(x) * m).astype(int), 0, m - 1)
    jx = np.clip(np.floor((x - x.min()) / np.ptp(x) * m).astype(int), 0, m - 1)
    Lg = (jg[:, None] == jx[None, :]).astype(float)
    Lg = Lg / Lg.sum(1, keepdims=True)
    # a step function: the band is a union of m independent intervals, c from the max of m |N(0,1)|
    c = stats.norm.ppf((1 + 0.95 ** (1 / m)) / 2)
    res["regressogram"] = dict(par=m, L=L, Lg=Lg, c=c, kappa0=np.nan, cv=cvm.min())
    hs = log_grid(0.08, 1.5, 60)
    for name, meth in (("kernel", "nw"), ("local linear", "ll")):
        sel = select_h(x, y, hs, meth)
        h = sel["hbest"]
        L = smoother_matrix(x, h, meth)
        Lg = nw_matrix(x, xg, h) if meth == "nw" else ll_matrix(x, xg, h)
        k0 = tube_kappa0(Lg, xg)
        res[name] = dict(par=h, L=L, Lg=Lg, c=tube_c(k0), kappa0=k0, cv=sel["cv"].min())
    ssel = spline_select(x, y, log_grid(1e-7, 10, 60))
    ss = ssel["fit"]
    Lg = ss.weights_at(xg)
    k0 = tube_kappa0(Lg, xg)
    res["smoothing spline"] = dict(par=ss.df, L=ss.L, Lg=Lg, c=tube_c(k0), kappa0=k0, cv=ssel["cv"].min())
    for k, v in res.items():
        v["sigma"] = math.sqrt(sigma2_linear(v["L"], y))
        v["df"] = float(np.trace(v["L"]))
        fit = v["Lg"] @ y
        se = v["sigma"] * np.linalg.norm(v["Lg"], axis=1)
        v["fit"], v["lo"], v["up"] = fit, fit - v["c"] * se, fit + v["c"] * se
    return x, y, xg, res


def fig_5_S4(out=None):
    """Exercise 3 (glass): four smoothers tuned by cross-validation with 95 percent bands."""
    x, y, xg, res = glass_four()
    fig, ax = common.panel_grid(2, 2, size=(3.3, 2.5))
    for a, (k, v) in zip(ax.ravel(), res.items()):
        a.plot(x, y, ".", ms=3, color=GREY)
        a.fill_between(xg, v["lo"], v["up"], color=BLUE, alpha=.25, lw=0)
        a.plot(xg, v["fit"], color=RED, lw=1.4, drawstyle="steps-mid" if k == "regressogram" else "default")
        lab = "m = %d" % v["par"] if k == "regressogram" else ("%.1f df" % v["par"] if k == "smoothing spline" else "h = %.3f" % v["par"])
        a.set_title("%s, %s, c = %.2f" % (k, lab, v["c"]), fontsize=9)
        a.set_ylim(-8, 16)
        _style(a, "aluminium", "refractive index")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S4", out)


def fig_5_S5(out=None):
    """Exercise 3 (glass): effective kernels l(x) of the four smoothers at three values of x."""
    x, y, xg, res = glass_four()
    pts = [0.6, 1.4, 2.8]
    fig, ax = common.panel_grid(1, 3, size=(2.6, 2.4))
    cols = dict(zip(res, [GREY, RED, BLUE, GREEN]))
    for a, p in zip(ax[0], pts):
        g = int(np.argmin(np.abs(xg - p)))
        for k, v in res.items():
            a.plot(x, v["Lg"][g], color=cols[k], lw=1.2, marker=".", ms=2, label=k)
        a.axhline(0, color=GREY, lw=.5)
        a.set_title("x = %.1f" % p, fontsize=9.5)
        a.set_xlim(max(0, p - 1.2), p + 1.2)
        _style(a, r"$x_i$ (aluminium)", r"$\ell_i(x)$" if p == pts[0] else None)
    ax[0, 0].legend(fontsize=6.5, loc="upper left")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S5", out)


def fig_5_S6(out=None):
    """Exercise 10: B-spline bases of order M = 1, ..., 5 with five equally spaced knots on (0, 1)."""
    xx = np.linspace(0, 1, 801)
    fig, ax = common.panel_grid(1, 5, size=(1.9, 1.9))
    cols = [RED, BLUE, GREEN, MAGENTA, ORANGE, PURPLE, GREY, "#8c564b", "#17becf", "#bcbd22"]
    for j, M in enumerate(range(1, 6)):
        tau = augmented_knots(np.linspace(0, 1, 7)[1:-1], 0, 1, M)
        B = bspline_basis(xx, tau, M)
        for i in range(B.shape[1]):
            ax[0, j].plot(xx, B[:, i], color=cols[i % len(cols)], lw=1.1)
        ax[0, j].set_title("M = %d: %d functions" % (M, B.shape[1]), fontsize=8.5)
        ax[0, j].set_ylim(-0.05, 1.1)
        ax[0, j].tick_params(labelsize=7)
    fig.tight_layout()
    savefig(fig, CH, "fig5-S6", out)


def motor_fits():
    """Exercises 4 and 11: local linear (CV), cubic regression spline (number of knots by CV) and
    smoothing spline (CV) for the motorcycle data."""
    x, y = motor()
    o = np.argsort(x, kind="stable")
    x, y = x[o], y[o]
    xg = np.linspace(x.min(), x.max(), 500)
    sel = select_h(x, y, log_grid(0.8, 10, 60), "ll")
    ks = np.arange(1, 31)
    cvk = np.array([loo_cv(regression_spline(x, y, k), y) for k in ks])
    k = int(ks[np.argmin(cvk)])
    _, Hg = regression_spline(x, y, k, xg)
    ssel = spline_select(x, y, log_grid(1e-9, 1, 80))
    return dict(x=x, y=y, xg=xg, h=sel["hbest"], llg=ll_matrix(x, xg, sel["hbest"]) @ y, ll_df=sel["df"][sel["best"]],
                ks=ks, cvk=cvk, k=k, rs=Hg @ y, ss=ssel["fit"], sscv=ssel)


def fig_5_S7(out=None):
    """Exercises 4 and 11 (motorcycle): local linear, regression spline, smoothing spline; CV over knots."""
    D = motor_fits()
    fig, ax = common.panel_grid(1, 2, size=(3.4, 2.7))
    a = ax[0, 0]
    a.plot(D["x"], D["y"], ".", ms=3, color=GREY)
    a.plot(D["xg"], D["llg"], color=RED, label="local linear, h = %.2f" % D["h"])
    a.plot(D["xg"], D["rs"], color=BLUE, ls="--", label="regression spline, %d knots" % D["k"])
    a.plot(D["xg"], D["ss"](D["xg"]), color=GREEN, ls=":", lw=2, label="smoothing spline, %.1f df" % D["ss"].df)
    a.legend(fontsize=7, loc="lower right")
    _style(a, "time (ms)", "acceleration (g)")
    b = ax[0, 1]
    b.plot(D["ks"], D["cvk"], "o-", ms=3, color=BLUE)
    b.axvline(D["k"], color=RED, ls="--", lw=1)
    b.set_ylim(400, 1500)
    _style(b, "number of interior knots", "CV score")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S7", out)


def doppler_three(sigmas=(0.1, 1.0, 3.0)):
    """Exercises 12 and 13: local linear and smoothing spline fits with 95 percent bands."""
    out = []
    for s in sigmas:
        x, y = doppler_data(1000, s)
        sel = select_h(x, y, log_grid(0.002, 0.3, 60), "ll")
        h = sel["hbest"]
        L = ll_matrix(x, x, h)
        xg = np.linspace(0, 1, 800)
        Lg = ll_matrix(x, xg, h)
        k0 = tube_kappa0(Lg, xg)
        c = tube_c(k0)
        sg = math.sqrt(sigma2_linear(L, y))
        fit = Lg @ y
        se = sg * np.linalg.norm(Lg, axis=1)
        ssel = spline_select(x, y, log_grid(1e-12, 1e-1, 30))
        ss = ssel["fit"]
        Sg = ss.weights_at(xg)
        k1 = tube_kappa0(Sg, xg)
        c1 = tube_c(k1)
        s1 = math.sqrt(sigma2_linear(ss.L, y))
        f1 = Sg @ y
        se1 = s1 * np.linalg.norm(Sg, axis=1)
        out.append(dict(sigma=s, x=x, y=y, xg=xg, h=h, df=np.trace(L), c=c, kappa0=k0, fit=fit, lo=fit - c * se, up=fit + c * se,
                        sdf=ss.df, c1=c1, kappa1=k1, f1=f1, lo1=f1 - c1 * se1, up1=f1 + c1 * se1, sel=sel, ssel=ssel))
    return out


def fig_5_S8(out=None):
    """Exercises 12 and 13: Doppler data at sigma = 0.1, 1, 3; local linear and smoothing spline with bands."""
    R = doppler_three()
    fig, ax = common.panel_grid(3, 3, size=(2.6, 2.0))
    for i, D in enumerate(R):
        ax[i, 0].plot(D["x"], D["y"], ".", ms=1.2, color=GREY)
        ax[i, 0].set_title("data, sigma = %g" % D["sigma"], fontsize=8.5)
        for j, (f, lo, up, t) in enumerate([(D["fit"], D["lo"], D["up"], "local linear, %.0f df" % D["df"]),
                                            (D["f1"], D["lo1"], D["up1"], "smoothing spline, %.0f df" % D["sdf"])]):
            a = ax[i, j + 1]
            a.fill_between(D["xg"], lo, up, color=BLUE, alpha=.3, lw=0)
            a.plot(D["xg"], f, color=RED, lw=.9)
            a.plot(D["xg"], doppler(D["xg"]), color="k", lw=.6, ls=":")
            a.set_title(t, fontsize=8.5)
            a.set_ylim(-1.2, 1.2)
        for a in ax[i]:
            a.tick_params(labelsize=7)
    fig.tight_layout()
    savefig(fig, CH, "fig5-S8", out)


def ozone_selectors():
    """Exercise 14: kernel (Nadaraya-Watson, tricube) regression of ozone on temperature with h from
    CV (5.35), GCV (5.36), Cp (5.37 with sigma2 from (5.86) at the CV fit) and the plug-in rule (5.138)."""
    d = airquality()
    x, y = d["Temp"].to_numpy(float), d["Ozone"].to_numpy(float)
    o = np.argsort(x, kind="stable")
    x, y = x[o], y[o]
    hs = log_grid(1.5, 25, 80)
    sel = select_h(x, y, hs, "nw")
    s2 = sigma2_linear(nw_matrix(x, x, sel["hbest"]), y)
    cp = np.array([cp_score(nw_matrix(x, x, h), y, s2) for h in hs])
    hp, C = plugin_h(x, y)
    res = {"CV": sel["hbest"], "GCV": float(hs[np.argmin(sel["gcv"])]), "Cp": float(hs[np.argmin(cp)]), "plug-in": hp}
    return x, y, hs, sel, cp, res


def fig_5_S9(out=None):
    """Exercise 14: ozone on temperature, kernel regression with four bandwidth selectors."""
    x, y, hs, sel, cp, res = ozone_selectors()
    xg = np.linspace(x.min(), x.max(), 400)
    fig, ax = common.panel_grid(1, 2, size=(3.4, 2.7))
    a = ax[0, 0]
    a.plot(x, y, ".", ms=4, color=GREY)
    for (k, h), c, ls in zip(res.items(), [RED, BLUE, GREEN, PURPLE], ["-", "--", ":", "-."]):
        a.plot(xg, nw_matrix(x, xg, h) @ y, color=c, ls=ls, lw=1.6, label="%s: h = %.2f" % (k, h))
    a.legend(fontsize=7)
    _style(a, "temperature (F)", "ozone (ppb)")
    b = ax[0, 1]
    ok = np.isfinite(sel["cv"])
    b.semilogx(hs[ok], sel["cv"][ok], color=RED, label="CV")
    b.semilogx(hs[ok], sel["gcv"][ok], color=BLUE, ls="--", label="GCV")
    b.semilogx(hs[ok], cp[ok], color=GREEN, ls=":", lw=2, label="Cp")
    b.axvline(res["plug-in"], color=PURPLE, ls="-.", lw=1)
    b.set_ylim(min(cp[ok].min(), sel["cv"][ok].min()) * 0.95, sel["cv"][ok].min() * 1.6)
    b.legend(fontsize=7)
    _style(b, "bandwidth h", "score")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S9", out)


def ozone_compare(seed=0):
    """Exercise 22: ozone on (Solar.R, Wind, Temp): five methods compared by 10-fold CV error."""
    from sklearn.model_selection import KFold
    d = airquality()
    X = d[["Solar.R", "Wind", "Temp"]].to_numpy(float)
    Y = d["Ozone"].to_numpy(float)
    mu, sd = X.mean(0), X.std(0)
    Z = (X - mu) / sd
    kf = list(KFold(10, shuffle=True, random_state=seed).split(Z))
    out = {}

    def cv_err(fit_predict):
        e = 0.0
        for tr, te in kf:
            e += np.sum((Y[te] - fit_predict(Z[tr], Y[tr], Z[te])) ** 2)
        return e / len(Y)
    hs = np.linspace(1.2, 5, 39)
    cvs = [loo_cv(loclin_nd(Z, Y, Z, h), Y) for h in hs]
    h3 = float(hs[int(np.nanargmin(cvs))])
    out["local linear (3-d)"] = (cv_err(lambda A, b, C: loclin_nd(A, b, C, h3) @ b), "h = %.2f" % h3)
    hadd = [1.5, 1.5, 1.5]

    def add_fp(A, b, C):
        al, R, hh, _ = backfit(A, b, h=hadd)
        pred = np.full(len(C), al)
        for j in range(A.shape[1]):
            part = b - al - R.sum(1) + R[:, j]
            fj = ll_matrix(A[:, j], C[:, j], hadd[j]) @ part
            fj0 = ll_matrix(A[:, j], A[:, j], hadd[j]) @ part
            pred += fj - fj0.mean()
        return np.nan_to_num(pred, nan=al)
    out["additive (backfitting)"] = (cv_err(add_fp), "h = 1.5 (standardised)")

    def ppr_fp(A, b, C):
        m, dirs, terms, rss = ppr(A, b, M=2, restarts=2, seed=seed)
        pred = np.full(len(C), m)
        e = b - m
        for k in range(len(dirs)):
            z = A @ dirs[k]
            f, pr, _ = _ss_gcv_fit(z, e)
            pred += np.nan_to_num(pr(C @ dirs[k]))
            e = e - f
        return pred
    out["projection pursuit (M = 2)"] = (cv_err(ppr_fp), "")

    def tree_fp(A, b, C):
        t, _, _ = tree_fit(A, b, seed=seed)
        return t.predict(C)
    out["regression tree"] = (cv_err(tree_fp), "")

    def mars_fp(A, b, C):
        pred, _, _, _ = mars(A, b, max_terms=11, degree=2)
        return pred(C)
    out["MARS (degree 2)"] = (cv_err(mars_fp), "")
    lin = lambda A, b, C: np.c_[np.ones(len(C)), C] @ np.linalg.lstsq(np.c_[np.ones(len(A)), A], b, rcond=None)[0]
    out["linear regression"] = (cv_err(lin), "")
    return out


def ozone_additive_bands(h=1.5, alpha=0.05):
    """Exercise 23: with fixed bandwidths backfitting is a linear smoother, so each component
    r_j = L_j Y and the tube band applies to it. L_j is found by backfitting the unit vectors."""
    d = airquality()
    X = d[["Solar.R", "Wind", "Temp"]].to_numpy(float)
    Y = d["Ozone"].to_numpy(float)
    Z = (X - X.mean(0)) / X.std(0)
    n, p = Z.shape
    Ls = [np.zeros((n, n)) for _ in range(p)]
    I = np.eye(n)
    for i in range(n):
        al, R, _, _ = backfit(Z, I[:, i], h=[h] * p, iters=200, tol=1e-12)
        for j in range(p):
            Ls[j][:, i] = R[:, j]
    al, R, _, _ = backfit(Z, Y, h=[h] * p, iters=200, tol=1e-12)
    Lfull = np.ones((n, n)) / n + sum(Ls)
    s2 = sigma2_linear(Lfull, Y)
    out = []
    for j in range(p):
        o = np.argsort(Z[:, j])
        Lj = Ls[j][o]
        k0 = tube_kappa0(Lj, Z[o, j] + np.arange(n) * 1e-9)
        c = tube_c(k0, alpha)
        se = math.sqrt(s2) * np.linalg.norm(Lj, axis=1)
        out.append(dict(x=X[o, j], f=Lj @ Y, lo=Lj @ Y - c * se, up=Lj @ Y + c * se, c=c, kappa0=k0, err=np.max(np.abs(Lj @ Y - R[o, j]))))
    return out, al, s2


def fig_5_S10(out=None):
    """Exercises 22 and 23: the additive model for ozone with 95 percent bands for each component."""
    B, al, s2 = ozone_additive_bands()
    names = ["solar radiation", "wind (mph)", "temperature (F)"]
    fig, ax = common.panel_grid(1, 3, size=(2.5, 2.3))
    for a, b, nm in zip(ax[0], B, names):
        a.fill_between(b["x"], b["lo"], b["up"], color=BLUE, alpha=.3, lw=0, step=None)
        a.plot(b["x"], b["f"], color=RED)
        a.axhline(0, color=GREY, lw=.5)
        a.set_title("c = %.2f" % b["c"], fontsize=9)
        _style(a, nm, "component (ppb)" if nm == names[0] else None)
    fig.tight_layout()
    savefig(fig, CH, "fig5-S10", out)


def drop_test_null(n=150, sigma=0.5, h1=0.15, h2=0.35, reps=2000, B=2000, seed=24):
    """Exercise 24: T = ||(L12 - L1) Y||^2 for H0: r(x1, x2) = r(x1). Under H0, T = ||A(r + eps)||^2
    with A = L12 - L1: a weighted sum of non-central chi-squares. The null is approximated by
    simulating Y* = r_hat(x1) + sigma_hat eps* (a parametric bootstrap). Returns the true null
    sample, the bootstrap sample from one data set, and the exact mean."""
    rng = np.random.default_rng(seed)
    X = rng.uniform(size=(n, 2))
    r = lambda t: np.sin(2 * np.pi * t)
    L12 = loclin_nd(X, None, X, h2)
    L1 = ll_matrix(X[:, 0], X[:, 0], h1)
    A = L12 - L1
    mean_exact = float(np.sum((A @ r(X[:, 0])) ** 2) + sigma ** 2 * np.sum(A * A))
    E = rng.standard_normal((reps, n)) * sigma
    Tnull = np.sum(((r(X[:, 0])[None, :] + E) @ A.T) ** 2, axis=1)
    y = r(X[:, 0]) + sigma * rng.standard_normal(n)
    rhat = L1 @ y
    shat = math.sqrt(sigma2_linear(L1, y))
    Es = rng.standard_normal((B, n)) * shat
    Tboot = np.sum(((rhat[None, :] + Es) @ A.T) ** 2, axis=1)
    return Tnull, Tboot, mean_exact, float(np.sum((A @ y) ** 2))


def fig_5_S11(out=None):
    """Exercise 24: the true null distribution of T and its parametric-bootstrap estimate."""
    Tnull, Tboot, mean_exact, Tobs = drop_test_null()
    fig, ax = common.panel_grid(1, 1, size=(5.0, 2.7))
    a = ax[0, 0]
    bins = np.linspace(0, np.quantile(np.r_[Tnull, Tboot], 0.995), 50)
    a.hist(Tnull, bins, density=True, histtype="stepfilled", color=BLUE, alpha=.35, label="null distribution (simulated)")
    a.hist(Tboot, bins, density=True, histtype="step", color=RED, lw=1.5, label="bootstrap estimate from one data set")
    a.axvline(mean_exact, color="k", ls=":", lw=1, label="exact null mean")
    a.legend(fontsize=7.5)
    _style(a, "T", "density")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S11", out)


def sizer_map(x, y, hs, xg, alpha=0.05, sigma=None):
    """SiZer (Section 5.11; Chaudhuri and Marron 1999): for each h, the local quadratic slope
    estimate r'_h(x) (5.143) and its standard error sigma ||l(x, 1)||; a slope is flagged significant
    when |r'_h| > c se, with c from the tube formula for the slope process at that h (Exercise 21).
    sigma defaults to (5.86) at the cross-validated local linear fit. Returns a map with +1
    (increasing), -1 (decreasing), 0 (not significant), nan (fewer than 5 points in the window)."""
    if sigma is None:
        h0 = select_h(x, y, log_grid(np.ptp(x) / 100, np.ptp(x), 60), "ll")["hbest"]
        sigma = math.sqrt(sigma2_linear(ll_matrix(x, x, h0), y))
    S = np.full((len(hs), len(xg)), np.nan)
    for k, h in enumerate(hs):
        D = locpoly_matrix(x, xg, h, 2, deriv=1)
        ok = np.all(np.isfinite(D), axis=1) & (np.sum(kernel((xg[:, None] - x[None, :]) / h) > 0, axis=1) >= 5)
        if ok.sum() < 3:
            continue
        # the tube constant over the runs of consecutive valid grid points
        idx = np.nonzero(ok)[0]
        runs = np.split(idx, np.nonzero(np.diff(idx) > 1)[0] + 1)
        k0 = sum(tube_kappa0(D[r_], xg[r_]) for r_ in runs if len(r_) > 2)
        c = tube_c(k0, alpha) if k0 > 0 else stats.norm.ppf(1 - alpha / 2)
        d = D[ok] @ y
        se = sigma * np.linalg.norm(D[ok], axis=1)
        S[k, ok] = np.where(d > c * se, 1, np.where(d < -c * se, -1, 0))
    return S


def fig_5_S12(out=None):
    """Exercise 9 and Section 5.11: fits over a range of bandwidths and the SiZer map (glass data)."""
    x, y = glass()
    o = np.argsort(x, kind="stable")
    x, y = x[o], y[o]
    xg = np.linspace(x.min(), x.max(), 200)
    hs = log_grid(0.12, 2.0, 40)
    S = sizer_map(x, y, hs, xg)
    fig, ax = common.panel_grid(1, 2, size=(3.4, 2.7))
    a = ax[0, 0]
    a.plot(x, y, ".", ms=2.5, color=GREY)
    for h, c in zip([0.15, 0.3, 0.6, 1.2], [RED, ORANGE, BLUE, PURPLE]):
        a.plot(xg, ll_matrix(x, xg, h) @ y, color=c, lw=1.2, label="h = %g" % h)
    a.set_ylim(-6, 12)
    a.legend(fontsize=7)
    _style(a, "aluminium", "refractive index")
    from matplotlib.colors import ListedColormap
    b = ax[0, 1]
    cmap = ListedColormap([BLUE, "#d9d9d9", RED])
    b.pcolormesh(xg, np.log10(hs), S, cmap=cmap, vmin=-1.5, vmax=1.5, shading="auto")
    b.set_yticks(np.log10([0.15, 0.3, 0.6, 1.2]))
    b.set_yticklabels(["0.15", "0.3", "0.6", "1.2"])
    _style(b, "aluminium", "bandwidth h (log scale)")
    b.set_title("SiZer: red up, blue down, grey neither", fontsize=8.5)
    fig.tight_layout()
    savefig(fig, CH, "fig5-S12", out)


def fig_5_S13(out=None):
    """Exercise 18: local linear median (quantile) regression and the local linear mean, glass data."""
    x, y = glass()
    o = np.argsort(x, kind="stable")
    x, y = x[o], y[o]
    xg = np.linspace(x.min(), x.max(), 300)
    h = select_h(x, y, log_grid(0.08, 1.5, 60), "ll")["hbest"]
    fig, ax = common.panel_grid(1, 1, size=(5.0, 2.8))
    a = ax[0, 0]
    a.plot(x, y, ".", ms=3, color=GREY)
    a.plot(xg, ll_matrix(x, xg, h) @ y, color=RED, label="local linear mean, h = %.2f" % h)
    a.plot(xg, robust_local(x, y, xg, h, rho="quantile", alpha=0.5), color=BLUE, ls="--", label="local linear median")
    a.plot(xg, robust_local(x, y, xg, h, rho="huber"), color=GREEN, ls=":", lw=2, label="local linear Huber (c = 1.345)")
    a.legend(fontsize=7.5)
    a.set_ylim(-6, 12)
    _style(a, "aluminium", "refractive index")
    fig.tight_layout()
    savefig(fig, CH, "fig5-S13", out)



# =========================================================================== data for the browser demos
def export_json():
    """../data/ch05-regression.json: the data sets of the three demonstrations."""
    os.makedirs(DATA_OUT, exist_ok=True)
    xc, yc, se = cmb()
    xl, yl = lidar()
    xm, ym = motor()
    xg, yg = glass()
    o = np.argsort(xg, kind="stable")
    xd, yd = doppler_data(1000, 0.1)
    xb, yb, _, _ = bumps_analysis()
    r = lambda v: [float(t) for t in v]
    D = {"cmb": {"x": r(xc), "y": r(yc), "se": r(se)},
         "lidar": {"x": r(xl), "y": r(yl)},
         "motor": {"x": r(xm), "y": r(ym)},
         "glass": {"x": r(xg[o]), "y": r(yg[o])},
         "doppler": {"x": r(xd), "y": r(yd)},
         "bumps": {"x": r(xb), "y": r(yb)}}
    with open(os.path.join(DATA_OUT, "ch05-regression.json"), "w") as f:
        json.dump(D, f, separators=(",", ":"))
    print("wrote ../data/ch05-regression.json")


def _with_export(fun):
    def wrapped(out_dir=None):
        fun(out_dir)
        export_json()
    wrapped.__doc__ = fun.__doc__
    return wrapped


FIGURES = {
    "5.1": fig_5_1, "5.2": fig_5_2, "5.3": fig_5_3, "5.4": fig_5_4, "5.5": fig_5_5, "5.6": fig_5_6,
    "5.7": fig_5_7, "5.8": fig_5_8, "5.9": fig_5_9, "5.10": fig_5_10, "5.11": fig_5_11, "5.12": fig_5_12,
    "5.13": fig_5_13, "5.14": fig_5_14, "5.15": fig_5_15, "5.16": fig_5_16, "5.17": fig_5_17, "5.19": fig_5_19,
    "5.S1": fig_5_S1, "5.S2": fig_5_S2, "5.S3": fig_5_S3, "5.S4": fig_5_S4, "5.S5": fig_5_S5, "5.S6": fig_5_S6,
    "5.S7": fig_5_S7, "5.S8": fig_5_S8, "5.S9": fig_5_S9, "5.S10": fig_5_S10, "5.S11": fig_5_S11,
    "5.S12": fig_5_S12, "5.S13": _with_export(fig_5_S13),
}

if __name__ == "__main__":
    common.run(FIGURES, CH)
