"""Murphy (2023), Chapter 17: Bayesian neural networks.

Companion code for Chapter 17 of K. P. Murphy, Probabilistic Machine Learning: Advanced Topics (MIT Press,
2023; December 2025 draft), pp. 647-680, and its supplement (Supplementary Section 17.1, pp. 181-182), written
from scratch with numpy, scipy, scikit-learn and PyTorch (CPU, double precision, one thread), to be read next to
the lecture notes (../ch17-bayesian-neural-networks.html).

Methods implemented (the notes use these functions by name)
* Priors (Section 17.2): random functions drawn from a Gaussian prior on an MLP (mlp_prior_functions, (17.2)-(17.9)),
  the output variance recursion behind Xavier and LeCun scaling (prior_output_variance, Filling the gap 17.2).
* Posteriors (Section 17.3): a functional PyTorch MLP on a flat parameter vector (FMLP), MC dropout (LeNet,
  fig_17_2), the generalised Gauss-Newton Laplace approximation and its linearised predictive (ggn_laplace),
  mean-field variational inference with the reparameterisation and local reparameterisation estimators (bbb_*),
  Hamiltonian Monte Carlo with dual-averaging step size (hmc_batched), deep ensembles, random-prior ensembles
  (random_prior_ensemble), bootstrap ensembles, a random-feature GP output layer with spectral normalisation (SNGP),
  stochastic weight averaging on a quadratic (sgd_quadratic), the Laplace bridge (laplace_bridge).
* Generalisation (Section 17.4): mode connectivity with a Bezier curve (fig_17_11), the Hessian spectrum and the
  effective dimensionality (17.36), evidence for structured and shuffled labels, Bayesian linear regression with
  a rank-deficient design, the McAllester-type PAC-Bayes bound, BMA against MAP under a shift orthogonal to the data.
* Online inference (Section 17.5): the extended Kalman filter for an MLP, global and decoupled (ekf_mlp), recursive
  least squares, sequential Laplace, assumed density filtering with the probit integral, the variational continual
  learning objective in the limit beta -> 0.
* Hierarchical Bayesian neural networks (Section 17.6): 18 rotated two-moons tasks, separate and hierarchical MLPs
  sampled by HMC in the non-centred parameterisation (fig_17_19).
* The browser demonstrations: the exact algorithms the pages run (demo_prior_run, demo_post_run, demo_shift_run)
  with a port of the demo kit's seeded generator (DKRng, mulberry32 with Box-Muller).

Data. Everything is simulated (scikit-learn make_moons for the two-moons data sets) except Figure 17.2, which uses
the MNIST digits (LeCun et al. 1998). Seeds are fixed. The posterior sampled by HMC uses N(0, 1) priors on all
weights and biases unless stated.

Figures. Reproduced here: 17.1, 17.2, 17.4, 17.7, 17.11, 17.15, 17.16, 17.17, 17.19. Skipped, with the reason:
  17.3, 17.18   diagrams of an MLP and of a hierarchical model (nothing computed);
  17.5, 17.6, 17.8, 17.10, 17.13   sketches and cartoons taken from other papers (SWA, the loss landscape, batch
                ensemble, flat and sharp minima, model families); the ideas are computed in 17.S2, 17.S5 and 17.S6;
  17.9          the Laplace bridge on ImageNet classes (needs a large pretrained network); the bridge itself is
                reproduced on Gaussian logits in 17.S1;
  17.12, 17.14  CIFAR-100 and CIFAR-10-C training of CNNs on a grid of widths and depths and a ResNet-20 under
                corruption (GPU scale); the same two ideas are shown on small problems in 17.S6 and 17.S4.
Figure 17.15 uses 200 training points instead of 1000 and 500 retained HMC draws instead of 3000, to fit in the time
budget. Figure 17.4 uses the 50 points of the book's example with our own MLP and sampler.
Supplementary figures for the gaps filled in the notes:
  17.S1  the Laplace bridge against Monte Carlo draws of the softmax of Gaussian logits;
  17.S2  constant-step SGD on a quadratic, the stationary Gaussian and the SWA average;
  17.S3  randomize-then-optimise ensembles for a linear model against the exact posterior;
  17.S4  the predictive standard deviation of MC dropout, an ensemble and the exact posterior as N grows;
  17.S5  BMA against MAP when the test inputs move along a direction the training data never explored;
  17.S6  the Hessian spectrum, the effective dimensionality and the test loss of MLPs of increasing width;
  17.S7  log evidence of structured against shuffled labels;
  17.S8  global against decoupled extended Kalman filters for the MLP of Figure 17.16.
Run `python3 ch17_bayesian_neural_networks.py --list`, `python3 ch17_bayesian_neural_networks.py` (all figures) or
`python3 ch17_bayesian_neural_networks.py --fig 17.1`. Files go to ../img/ch17/; the data used by the browser
demonstrations go to ../data/ch17-*.json.
"""
import json
import os
import sys
import time

os.environ.setdefault("OPENBLAS_NUM_THREADS", "1")

import numpy as np
from scipy import linalg, special, stats

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import common  # noqa: E402
from common import BLUE, GREEN, GREY, MAGENTA, ORANGE, PURPLE, RED, CYAN, plt, savefig, panel_label, sigmoid  # noqa: E402

CH = 17
HERE = os.path.dirname(os.path.abspath(__file__))
DATA_OUT = os.path.normpath(os.path.join(HERE, "..", "data"))
CACHE_DIR = os.path.join(HERE, "cache", "ch17-work")
os.makedirs(CACHE_DIR, exist_ok=True)


def _torch():
    return common.torch_setup(0, 1)


# =========================================================================== generic building blocks
class FMLP:
    """A multilayer perceptron whose parameters are one flat vector theta (weights then biases, layer by layer).

    widths = [d_in, h_1, ..., d_out]; theta may have leading batch dimensions (..., P); X has shape (N, d_in) or
    (..., N, d_in). All layers but the last use the activation `act`; the last layer is linear.
    """

    def __init__(self, widths, act="tanh"):
        self.widths = list(widths)
        self.act = act
        self.shapes = [(widths[i], widths[i + 1]) for i in range(len(widths) - 1)]
        self.P = int(sum(a * b + b for a, b in self.shapes))

    def split(self, th):
        out, k = [], 0
        for a, b in self.shapes:
            W = th[..., k:k + a * b].reshape(th.shape[:-1] + (a, b)); k += a * b
            bb = th[..., k:k + b].reshape(th.shape[:-1] + (1, b)); k += b
            out.append((W, bb))
        return out

    def activ(self, h):
        torch = _torch()
        if self.act == "tanh":
            return torch.tanh(h)
        if self.act == "relu":
            return torch.relu(h)
        if self.act == "sigmoid":
            return torch.sigmoid(h)
        raise ValueError(self.act)

    def __call__(self, th, X, drop=0.0, gen=None, features=False):
        torch = _torch()
        h = X
        L = len(self.shapes)
        for i, (W, b) in enumerate(self.split(th)):
            if features and i == L - 1:
                return h
            h = torch.matmul(h, W) + b
            if i < L - 1:
                h = self.activ(h)
                if drop > 0:
                    m = (torch.rand(h.shape, generator=gen, dtype=h.dtype) > drop).to(h.dtype) / (1 - drop)
                    h = h * m
        return h

    def init(self, rng, gain=1.0):
        """LeCun initialisation, weights N(0, gain^2 / fan_in), zero biases."""
        parts = []
        for a, b in self.shapes:
            parts.append(rng.normal(0, gain / np.sqrt(a), a * b))
            parts.append(np.zeros(b))
        return np.concatenate(parts)


def adam_fit(fun, th0, steps, lr=0.01, clip=None):
    """Minimise fun(theta) (a scalar torch function) with Adam from numpy th0; returns (theta numpy, final loss)."""
    torch = _torch()
    th = torch.tensor(np.asarray(th0, float), dtype=torch.float64, requires_grad=True)
    opt = torch.optim.Adam([th], lr=lr)
    loss = None
    for _ in range(steps):
        opt.zero_grad()
        loss = fun(th)
        loss.backward()
        opt.step()
    return th.detach().numpy().copy(), float(loss.detach())


def hmc_batched(logp, theta0, n_warm, n_samp, L, eps0, rng, delta=0.8, jitter=True, thin=1, verbose=False):
    """Hamiltonian Monte Carlo with unit mass matrix for B independent chains at once.

    logp(theta) takes a torch tensor (B, P) and returns the B log densities (the chains must not interact, so
    that summing and differentiating gives each chain its own gradient). The step size of each chain is adapted
    during warm-up by the dual averaging of Hoffman and Gelman (2014) to the target acceptance delta; the number
    of leapfrog steps is L (random in [L/2, L] when jitter). Returns (samples (n_samp, B, P), mean accept (B,)).
    """
    torch = _torch()
    th = torch.tensor(np.atleast_2d(np.asarray(theta0, float)), dtype=torch.float64)
    B, P = th.shape

    def vg(t):
        t = t.detach().requires_grad_(True)
        lp = logp(t)
        g, = torch.autograd.grad(lp.sum(), t)
        return lp.detach(), g

    lp, g = vg(th)
    eps = np.full(B, float(eps0))
    mu = np.log(10 * eps)
    Hbar = np.zeros(B); logeb = np.zeros(B)
    gam, t0, kap = 0.05, 10.0, 0.75
    out, acc_tot = [], np.zeros(B)
    for it in range(n_warm + n_samp):
        e = torch.tensor(eps * (1 + 0.2 * (rng.uniform(size=B) - 0.5)) if jitter else eps, dtype=torch.float64)[:, None]
        Ls = int(rng.integers(max(1, L // 2), L + 1)) if jitter else L
        p0 = torch.tensor(rng.normal(size=(B, P)), dtype=torch.float64)
        q, p, gq = th.clone(), p0 + 0.5 * e * g, g
        for s in range(Ls):
            q = q + e * p
            lpq, gq = vg(q)
            if s < Ls - 1:
                p = p + e * gq
        p = p + 0.5 * e * gq
        dH = (-lpq + 0.5 * (p ** 2).sum(1)) - (-lp + 0.5 * (p0 ** 2).sum(1))
        dH = dH.numpy()
        alpha = np.where(np.isfinite(dH), np.minimum(1.0, np.exp(-np.where(np.isfinite(dH), dH, 1e9))), 0.0)
        ok = rng.uniform(size=B) < alpha
        okt = torch.tensor(ok)[:, None]
        th = torch.where(okt, q, th)
        lp = torch.where(torch.tensor(ok), lpq, lp)
        g = torch.where(okt, gq, g)
        if it < n_warm:
            m = it + 1
            Hbar = (1 - 1 / (m + t0)) * Hbar + (delta - alpha) / (m + t0)
            logeps = mu - np.sqrt(m) / gam * Hbar
            eta = m ** (-kap)
            logeb = eta * logeps + (1 - eta) * logeb
            eps = np.exp(logeps)
            if it == n_warm - 1:
                eps = np.exp(logeb)
        else:
            acc_tot += alpha
            if (it - n_warm) % thin == 0:
                out.append(th.numpy().copy())
        if verbose and it % 200 == 0:
            print("  hmc it", it, "eps", eps.mean(), "acc", alpha.mean(), flush=True)
    return np.array(out), acc_tot / n_samp


def make_moons_np(n, noise, seed):
    from sklearn.datasets import make_moons
    X, y = make_moons(n_samples=n, noise=noise, random_state=seed)
    return X, y.astype(float)


# =========================================================================== the demo kit's generator
class DKRng:
    """Exact port of DK.rng in ../../demo-kit.js (mulberry32 uniforms; Box-Muller normals with a spare)."""

    def __init__(self, seed):
        self.a = seed & 0xFFFFFFFF
        self.spare = None

    @staticmethod
    def _imul(x, y):
        return (x * y) & 0xFFFFFFFF

    def u(self):
        M = 0xFFFFFFFF
        self.a = (self.a + 0x6D2B79F5) & M
        a = self.a
        t = self._imul(a ^ (a >> 15), 1 | a)
        t = ((t + self._imul(t ^ (t >> 7), 61 | t)) & M) ^ t
        return ((t ^ (t >> 14)) & M) / 4294967296.0

    def pos(self):
        x = 0.0
        while x == 0.0:
            x = self.u()
        return x

    def norm(self):
        if self.spare is not None:
            s, self.spare = self.spare, None
            return s
        r = np.sqrt(-2 * np.log(self.pos()))
        th = 2 * np.pi * self.u()
        self.spare = r * np.sin(th)
        return r * np.cos(th)


# =========================================================================== 17.2 priors
def mlp_activation(a, act):
    if act == "sigmoid":
        return 1.0 / (1.0 + np.exp(-a))
    if act == "tanh":
        return np.tanh(a)
    if act == "relu":
        return np.maximum(a, 0.0)
    raise ValueError(act)


def mlp_prior_functions(x, a1, b1, a2, b2, H=12, n=10, seed=0, act="sigmoid"):
    """n random functions f(x) = W2 phi(W1 x + b1) + b2 with W1 = a1 eta, b1 = b1 eps, ... as in (17.8)-(17.9)
    (scalar input and output, H hidden units), drawn with the demo kit's generator. Returns (n, len(x))."""
    r = DKRng(seed)
    out = []
    for _ in range(n):
        W1 = np.array([a1 * r.norm() for _ in range(H)])
        B1 = np.array([b1 * r.norm() for _ in range(H)])
        W2 = np.array([a2 * r.norm() for _ in range(H)])
        B2 = b2 * r.norm()
        h = mlp_activation(np.outer(x, W1) + B1, act)
        out.append(h @ W2 + B2)
    return np.array(out)


def prior_output_variance(x, a1, b1, a2, b2, H, act="sigmoid"):
    """Exact prior variance of f(x) under (17.5): H a2^2 E[phi(z)^2] + b2^2 with z ~ N(0, a1^2 x^2 + b1^2)."""
    from scipy import integrate
    s = np.sqrt(a1 ** 2 * x ** 2 + b1 ** 2)
    if act == "relu":
        m2 = s ** 2 / 2
    elif s == 0:
        m2 = float(mlp_activation(0.0, act) ** 2)
    else:
        m2 = integrate.quad(lambda t: stats.norm.pdf(t) * mlp_activation(s * t, act) ** 2, -12, 12, limit=400, epsabs=1e-13, epsrel=1e-13)[0]
    return H * a2 ** 2 * m2 + b2 ** 2


def fig_17_1(out=None):
    """Random functions from a Gaussian prior on a one-hidden-layer sigmoid MLP, four hyperparameter settings."""
    x = np.linspace(-1, 1, 201)
    sets = [(5, 1, 1, 1), (25, 1, 1, 1), (5, 5, 1, 1), (5, 1, 5, 1)]
    fig, axes = plt.subplots(1, 4, figsize=(12.5, 2.9), sharey=True)
    for k, (ax, (a1, b1, a2, b2)) in enumerate(zip(axes, sets)):
        F = mlp_prior_functions(x, a1, b1, a2, b2, H=12, n=10, seed=3)
        for f in F:
            ax.plot(x, f, color="k", lw=0.7)
        ax.set_ylim(-10, 10)
        ax.set_xlabel("$x$")
        ax.set_title(r"$\alpha_1=%g,\beta_1=%g,\alpha_2=%g,\beta_2=%g$" % (a1, b1, a2, b2), fontsize=8.5)
        panel_label(ax, "abcd"[k], "lower left")
    axes[0].set_ylabel("$y$")
    fig.tight_layout()
    return savefig(fig, CH, "fig17-1", out)


# =========================================================================== 17.3.1 Monte Carlo dropout
def _lenet(torch):
    nn = torch.nn

    class LeNet(nn.Module):
        def __init__(self, p=0.5):
            super().__init__()
            self.c1 = nn.Conv2d(1, 6, 5, padding=2)
            self.c2 = nn.Conv2d(6, 16, 5)
            self.f1 = nn.Linear(16 * 5 * 5, 120)
            self.f2 = nn.Linear(120, 84)
            self.f3 = nn.Linear(84, 10)
            self.p = p

        def forward(self, x, mc=False):
            x = torch.max_pool2d(torch.relu(self.c1(x)), 2)
            x = torch.max_pool2d(torch.relu(self.c2(x)), 2)
            x = x.flatten(1)
            x = torch.dropout(torch.relu(self.f1(x)), self.p, True if mc else self.training)
            x = torch.dropout(torch.relu(self.f2(x)), self.p, True if mc else self.training)
            return self.f3(x)

    return LeNet


def _rotate_digit(img, deg):
    from scipy import ndimage
    return ndimage.rotate(img.reshape(28, 28), deg, reshape=False, order=1)


def mc_dropout_rotated_digit(n_train=20000, epochs=4, n_mc=50, seed=0, digit=1):
    """Train a LeNet with dropout on MNIST, then return the logits and probabilities of n_mc stochastic forward
    passes for one test image of `digit` rotated by 0, 15, ..., 165 degrees."""
    torch = common.torch_setup(seed, 1)
    X, y = common.mnist(True)
    Xt, yt = common.mnist(False)
    rng = np.random.default_rng(seed)
    idx = rng.permutation(len(X))[:n_train]
    Xtr = torch.tensor(X[idx].reshape(-1, 1, 28, 28), dtype=torch.float32)
    ytr = torch.tensor(y[idx])
    net = _lenet(torch)()
    opt = torch.optim.Adam(net.parameters(), lr=2e-3)
    net.train()
    for ep in range(epochs):
        perm = torch.randperm(n_train)
        for i in range(0, n_train, 128):
            b = perm[i:i + 128]
            opt.zero_grad()
            loss = torch.nn.functional.cross_entropy(net(Xtr[b]), ytr[b])
            loss.backward(); opt.step()
    net.eval()
    with torch.no_grad():
        acc = (net(torch.tensor(Xt[:5000].reshape(-1, 1, 28, 28), dtype=torch.float32)).argmax(1).numpy() == yt[:5000]).mean()
    k = int(np.where(yt == digit)[0][3])
    angles = np.arange(0, 180, 15)
    imgs = np.stack([_rotate_digit(Xt[k], a) for a in angles])
    xin = torch.tensor(imgs.reshape(-1, 1, 28, 28), dtype=torch.float32)
    logits = []
    with torch.no_grad():
        for _ in range(n_mc):
            logits.append(net(xin, mc=True).numpy())
    logits = np.array(logits)
    return dict(angles=angles, imgs=imgs, logits=logits, probs=special.softmax(logits, axis=-1), test_acc=float(acc))


def fig_17_2(out=None):
    """MC dropout on a LeNet: a digit 1 rotated in steps of 15 degrees, logits and probabilities of three classes."""
    r = mc_dropout_rotated_digit()
    print("   LeNet test accuracy on 5000 test images: %.3f" % r["test_acc"])
    classes = [1, 5, 7]
    cols = [BLUE, RED, GREEN]
    fig = plt.figure(figsize=(10.5, 4.0))
    gs = fig.add_gridspec(2, 2, height_ratios=[0.5, 2.0], hspace=0.3, wspace=0.28)
    gt = gs[0, :].subgridspec(1, 12, wspace=0.08)
    for j in range(12):
        ax = fig.add_subplot(gt[0, j])
        ax.imshow(r["imgs"][j], cmap="gray_r"); ax.set_xticks([]); ax.set_yticks([])
        ax.set_title("%d" % (j + 1), fontsize=7.5, pad=1)
    for c, (key, lab) in enumerate([("logits", "logits (softmax inputs)"), ("probs", "probabilities (softmax outputs)")]):
        ax = fig.add_subplot(gs[1, c])
        v = r[key]
        for cl, col in zip(classes, cols):
            m, s = v[:, :, cl].mean(0), v[:, :, cl].std(0)
            ax.errorbar(np.arange(1, 13), m, yerr=s, color=col, marker="o", ms=3, lw=1.0, capsize=2, label="class %d" % cl)
        ax.set_xlabel("rotation index (15 degrees per step)"); ax.set_ylabel(lab)
        ax.set_xticks(range(1, 13)); ax.set_title("(%s)" % "ab"[c], fontsize=9, loc="left")
    ax.legend(loc="center right", fontsize=7.5)
    fig.suptitle("")
    return savefig(fig, CH, "fig17-2", out)


# =========================================================================== 17.3.7 HMC for an MLP classifier
def bernoulli_mlp_logpost(net, X, y, prior_sd=1.0):
    """Returns logp(theta (B, P)) = sum_n [y_n z_n - softplus(z_n)] - |theta|^2 / (2 prior_sd^2) for an MLP with
    one logit output, with X (N, d) shared by the chains or (B, N, d) per chain."""
    torch = _torch()
    Xt = torch.tensor(X, dtype=torch.float64)
    yt = torch.tensor(y, dtype=torch.float64)

    def logp(th):
        z = net(th, Xt)[..., 0]
        ll = (yt * z - torch.nn.functional.softplus(z)).sum(-1)
        return ll - 0.5 * (th ** 2).sum(-1) / prior_sd ** 2
    return logp


def _moons_grid(lo=(-2.5, -1.8), hi=(3.5, 2.2), n=80):
    gx = np.linspace(lo[0], hi[0], n)
    gy = np.linspace(lo[1], hi[1], n)
    xx, yy = np.meshgrid(gx, gy)
    return gx, gy, np.c_[xx.ravel(), yy.ravel()]


def hmc_moons(seed=314, n_data=50, noise=0.2, widths=(2, 8, 8, 1), n_warm=600, n_samp=400, L=30):
    """Fit an MLP to two moons with HMC; returns grid, mean and std of p(y=1|x) over the draws, data."""
    torch = _torch()
    X, y = make_moons_np(n_data, noise, seed)
    net = FMLP(widths, "tanh")
    rng = np.random.default_rng(seed)
    th0 = net.init(rng)
    # a short optimisation first, so that HMC starts near a mode
    Xt = torch.tensor(X); yt = torch.tensor(y)
    th0, _ = adam_fit(lambda t: -(yt * net(t, Xt)[..., 0] - torch.nn.functional.softplus(net(t, Xt)[..., 0])).sum() + 0.5 * (t ** 2).sum(), th0, 800, 0.02)
    S, acc = hmc_batched(bernoulli_mlp_logpost(net, X, y), th0, n_warm, n_samp, L, 0.05, rng)
    gx, gy, G = _moons_grid()
    with torch.no_grad():
        Z = net(torch.tensor(S[:, 0, :]), torch.tensor(G))[..., 0].numpy()
    P = sigmoid(Z)
    return dict(X=X, y=y, G=G, gx=gx, gy=gy, mean=P.mean(0), std=P.std(0), acc=float(acc.mean()))


def fig_17_4(out=None):
    """An MLP fitted to two moons by HMC: posterior mean and standard deviation of p(y=1|x)."""
    r = hmc_moons()
    n = len(r["gx"])
    fig, axes = plt.subplots(1, 2, figsize=(9.2, 3.5))
    for ax, key, title, cm in [(axes[0], "mean", "posterior mean of $p(y=1\\mid x)$", "RdBu_r"), (axes[1], "std", "posterior std of $p(y=1\\mid x)$", "viridis")]:
        im = ax.contourf(r["gx"], r["gy"], r[key].reshape(n, n), levels=20, cmap=cm)
        ax.scatter(*r["X"][r["y"] == 0].T, c="k", s=14, marker="o", edgecolor="w", lw=0.4)
        ax.scatter(*r["X"][r["y"] == 1].T, c="w", s=14, marker="^", edgecolor="k", lw=0.5)
        ax.set_title(title); ax.set_xlabel("$x_1$")
        fig.colorbar(im, ax=ax, fraction=0.046)
    axes[0].set_ylabel("$x_2$")
    panel_label(axes[0], "a", "upper left"); panel_label(axes[1], "b", "upper left")
    fig.tight_layout()
    print("   HMC acceptance %.2f" % r["acc"])
    return savefig(fig, CH, "fig17-4", out)


# =========================================================================== 17.3.9.2 random-prior ensembles
def _reg_data_random_priors():
    x = np.linspace(0.0, 0.5, 9)
    y = 0.5 + 0.35 * np.sin(2 * np.pi * x * 1.6)
    return x[:, None], y[:, None]


def random_prior_ensemble(beta, M=9, seed=0, steps=1200, hidden=32, prior_scale=0.02, X=None, Y=None, xg=None):
    """Ensemble of M members g_i(x) = t_i(x) + beta p_i(x) (17.20): p_i a fixed random MLP (output scale
    prior_scale), t_i a trainable MLP fitted to the data by Adam. Returns prior, trainable and total curves on xg."""
    torch = _torch()
    if X is None:
        X, Y = _reg_data_random_priors()
    xg = np.linspace(-0.5, 1.0, 301)[:, None] if xg is None else xg
    net = FMLP([1, hidden, 1], "tanh")
    Xt, Yt, Xg = (torch.tensor(a, dtype=torch.float64) for a in (X, Y, xg))
    P, T, G = [], [], []
    for i in range(M):
        rng = np.random.default_rng(1000 * seed + i)
        thp = torch.tensor(net.init(rng, gain=1.0))
        # rescale the output layer so that p_i has standard deviation about prior_scale
        thp = thp.clone()
        k = hidden + hidden  # first-layer weights and biases
        thp[k:k + hidden] *= prior_scale
        thp[k + hidden:] = 0.0
        with torch.no_grad():
            pt = net(thp, Xt); pg = net(thp, Xg)
        th0 = net.init(np.random.default_rng(1000 * seed + 500 + i), gain=0.3)
        th, _ = adam_fit(lambda t: ((Yt - net(t, Xt) - beta * pt) ** 2).sum(), th0, steps, 0.03)
        with torch.no_grad():
            tg = net(torch.tensor(th), Xg)
        P.append((beta * pg).numpy()[:, 0]); T.append(tg.numpy()[:, 0]); G.append((tg + beta * pg).numpy()[:, 0])
    return dict(x=xg[:, 0], prior=np.array(P), train=np.array(T), total=np.array(G))


def fig_17_7(out=None):
    """Deep ensembles with random priors: members for beta = 3 and the ensemble for four values of beta."""
    X, Y = _reg_data_random_priors()
    R3 = random_prior_ensemble(50.0, M=8)
    betas = [0.001, 5, 50, 100]
    fig = plt.figure(figsize=(11.5, 7.2))
    outer = fig.add_gridspec(1, 2, wspace=0.15, width_ratios=[1.1, 1])
    ga = outer[0].subgridspec(4, 2, hspace=0.45, wspace=0.12)
    for i in range(8):
        ax = fig.add_subplot(ga[i // 2, i % 2])
        ax.plot(R3["x"], R3["prior"][i], color=BLUE, lw=1.0)
        ax.plot(R3["x"], R3["train"][i], color=ORANGE, lw=1.0)
        ax.plot(R3["x"], R3["total"][i], color=GREEN, lw=1.3)
        ax.plot(X[:, 0], Y[:, 0], "k.", ms=3.5)
        ax.set_ylim(-0.4, 1.3); ax.set_xlim(-0.5, 1)
        ax.text(0.04, 0.9, "model #%d" % (i + 1), transform=ax.transAxes, fontsize=7.5, va="top")
        if i % 2: ax.set_yticklabels([])
        if i < 6: ax.set_xticklabels([])
    gb = outer[1].subgridspec(4, 2, hspace=0.45, wspace=0.12)
    for r_, b in enumerate(betas):
        R = random_prior_ensemble(b, M=9)
        m, s = R["total"].mean(0), R["total"].std(0)
        ax = fig.add_subplot(gb[r_, 0])
        ax.fill_between(R["x"], m - 2 * s, m + 2 * s, color=RED, alpha=0.25, lw=0)
        ax.plot(R["x"], m, color=RED, lw=1.3); ax.plot(X[:, 0], Y[:, 0], "k.", ms=3.5)
        ax.set_ylim(-1.5, 2.5); ax.set_xlim(-0.5, 1)
        ax.text(0.04, 0.92, r"$\beta=%g$" % b, transform=ax.transAxes, fontsize=8, va="top")
        ax2 = fig.add_subplot(gb[r_, 1])
        for g in R["total"]:
            ax2.plot(R["x"], g, color=GREY, lw=0.7)
        ax2.plot(X[:, 0], Y[:, 0], "k.", ms=3.5); ax2.set_ylim(-1.5, 2.5); ax2.set_xlim(-0.5, 1); ax2.set_yticklabels([])
        if r_ < 3: ax.set_xticklabels([]); ax2.set_xticklabels([])
    fig.text(0.27, 0.93, "(a) members: prior (blue), trainable (orange), sum (green), $\\beta=50$", ha="center", fontsize=9)
    fig.text(0.73, 0.93, "(b) ensemble mean $\\pm2$ sd (left) and members (right)", ha="center", fontsize=9)
    return savefig(fig, CH, "fig17-7", out)


# =========================================================================== 17.4.2 mode connectivity (Figure 17.11)
def _conn_data(seed=0):
    rng = np.random.default_rng(seed)
    x = np.r_[rng.uniform(-2.4, -0.7, 14), rng.uniform(0.7, 2.4, 14)]
    y = np.sin(1.8 * x) + 0.3 * x + 0.08 * rng.normal(size=x.size)
    return x[:, None], y[:, None]


def mode_connectivity(seed=0, steps=6000, curve_steps=3000, widths=(1, 40, 40, 1), wd=1e-5):
    """Two independently trained MLPs on a 1-d regression problem with a gap, a quadratic Bezier curve between
    them trained to keep the loss low along its length (Garipov et al. 2018), and the loss on the plane through the
    two modes and the mid-point of the curve."""
    torch = _torch()
    X, Y = _conn_data(seed)
    net = FMLP(list(widths), "tanh")
    Xt, Yt = torch.tensor(X), torch.tensor(Y)

    def loss(t):
        return ((net(t, Xt) - Yt) ** 2).mean(-2).sum(-1) / 2 + wd * (t ** 2).sum(-1) / 2
    modes = []
    for s in (1, 2):
        th0 = net.init(np.random.default_rng(100 * seed + s), gain=3.0)
        th, _ = adam_fit(lambda t: loss(t), th0, steps, 0.005)
        modes.append(th)
    t1, t2 = (torch.tensor(m) for m in modes)
    phi = ((t1 + t2) / 2).clone().requires_grad_(True)
    opt = torch.optim.Adam([phi], lr=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(curve_steps):
        t = torch.rand(8, 1, generator=g, dtype=torch.float64)
        pts = (1 - t) ** 2 * t1 + 2 * t * (1 - t) * phi + t ** 2 * t2
        opt.zero_grad(); loss(pts).mean().backward(); opt.step()
    phi = phi.detach()
    bez = lambda t: (1 - t) ** 2 * t1 + 2 * t * (1 - t) * phi + t ** 2 * t2
    return dict(net=net, X=X, Y=Y, t1=t1, t2=t2, phi=phi, bez=bez, loss=loss)


def fig_17_11(out=None):
    """Diversity of functions along a low-loss curve connecting two modes, and the loss on a plane through them."""
    torch = _torch()
    r = mode_connectivity()
    net, bez, loss = r["net"], r["bez"], r["loss"]
    xg = torch.linspace(-4.5, 4.5, 300, dtype=torch.float64)[:, None]
    ts = [0.0, 0.33, 0.67, 1.0]
    fig = plt.figure(figsize=(10.5, 6.4))
    gs = fig.add_gridspec(2, 4, height_ratios=[1, 1.5], hspace=0.4, wspace=0.85)
    with torch.no_grad():
        for k, t in enumerate(ts):
            ax = fig.add_subplot(gs[0, k])
            ax.plot(xg[:, 0], net(bez(torch.tensor(t)), xg)[:, 0], color=BLUE, lw=1.4)
            ax.plot(r["X"][:, 0], r["Y"][:, 0], "k.", ms=4)
            ax.set_ylim(-4.5, 4.5); ax.set_title("$t=%g$ on the curve" % t, fontsize=9); ax.set_xlabel("$x$")
    # plane through t1, t2 and the curve mid-point
    t1, t2, phi = r["t1"], r["t2"], r["phi"]
    mid = bez(torch.tensor(0.5))
    e1 = (t2 - t1); L1 = e1.norm(); e1 = e1 / L1
    v = mid - t1; v = v - (v @ e1) * e1; L2 = v.norm(); e2 = v / L2
    tt = torch.linspace(0, 1, 101, dtype=torch.float64)[:, None]
    path = bez(tt)
    pc = lambda th: np.c_[((th - t1) @ e1).numpy(), ((th - t1) @ e2).numpy()]
    cp = pc(path)
    u = np.linspace(-0.35 * L1.item(), 1.35 * L1.item(), 70)
    w = np.linspace(-0.6 * L2.item() - 0.5, 1.9 * L2.item() + 0.5, 70)
    UU, WW = np.meshgrid(u, w)
    pts = t1 + torch.tensor(UU.ravel())[:, None] * e1 + torch.tensor(WW.ravel())[:, None] * e2
    with torch.no_grad():
        Lg = loss(pts).numpy().reshape(UU.shape)
    ax = fig.add_subplot(gs[1, :2])
    cs = ax.contourf(UU, WW, np.log10(Lg), levels=np.linspace(np.log10(Lg.min()), min(np.log10(Lg.max()), 1.0), 18), cmap="viridis_r", extend="max")
    ax.plot(cp[:, 0], cp[:, 1], color="w", lw=2.0); ax.plot(cp[[0, -1], 0], cp[[0, -1], 1], "ro", ms=6)
    ax.plot([0, L1.item()], [0, 0], "w--", lw=1.0)
    fig.colorbar(cs, ax=ax, pad=0.02, label=r"$\log_{10}$ training loss")
    ax.set_xlabel("direction along the two modes"); ax.set_ylabel("orthogonal direction")
    ax.set_title("loss on the plane through the modes and the curve mid-point", fontsize=9)
    ax = fig.add_subplot(gs[1, 2:])
    with torch.no_grad():
        lc = loss(path).numpy(); ll = loss((1 - tt) * t1 + tt * t2).numpy()
    ax.semilogy(tt[:, 0], lc, color=BLUE, label="Bezier curve"); ax.semilogy(tt[:, 0], ll, color=RED, label="straight line")
    ax.set_xlabel("$t$"); ax.set_ylabel("training loss"); ax.legend()
    ax.set_title("loss along the curve and along the line", fontsize=9)
    print("   barrier: line max %.3g, curve max %.3g, modes %.3g %.3g" % (ll.max(), lc.max(), lc[0], lc[-1]))
    return savefig(fig, CH, "fig17-11", out)


# =========================================================================== 17.4.6.2 Figure 17.15: eight predictors
def moons_ood(n=200, noise=0.2, seed=6, n_ood=50):
    X, y = make_moons_np(n, noise, seed)
    rng = np.random.default_rng(seed)
    ood = np.array([-1.0, -2.0]) + 0.2 * rng.normal(size=(n_ood, 2))
    return X, y, ood


def _bce_loss(net, X, y, wd=0.0, drop=0.0, gen=None):
    torch = _torch()
    Xt, yt = torch.tensor(X), torch.tensor(y)

    def f(t):
        z = net(t, Xt, drop=drop, gen=gen)[..., 0]
        return (torch.nn.functional.softplus(z) - yt * z).mean() + wd * (t ** 2).sum()
    return f


def _grid15(n=90):
    gx = np.linspace(-2.2, 4.0, n); gy = np.linspace(-3.0, 2.5, n)
    xx, yy = np.meshgrid(gx, gy)
    return gx, gy, np.c_[xx.ravel(), yy.ravel()]


def sngp_fit(X, y, seed=0, width=64, depth=3, D=512, bound=1.0, steps=1500, lscale=1.5, lr=0.01):
    """Spectrally normalised GP (SNGP, Section 17.3.6): a residual MLP whose weight matrices are rescaled to have
    spectral norm at most `bound`, followed by a random-feature GP output layer (D cosine features of an RBF
    kernel) trained by MAP; the Laplace precision of the output weights gives the predictive variance of the
    logit, and the mean-field adjustment p = sigmoid(mu / sqrt(1 + pi var / 8)) is used. Returns predict(Xnew)."""
    torch = _torch()
    rng = np.random.default_rng(seed)
    W0 = torch.tensor(rng.normal(0, 1, (2, width)))                       # fixed random input layer
    Ws = [torch.tensor(rng.normal(0, 1 / np.sqrt(width), (width, width)), requires_grad=True) for _ in range(depth)]
    bs = [torch.zeros(width, dtype=torch.float64, requires_grad=True) for _ in range(depth)]
    R = torch.tensor(rng.normal(0, 1.0 / lscale, (width, D)))
    ph = torch.tensor(rng.uniform(0, 2 * np.pi, D))
    beta = torch.zeros(D, dtype=torch.float64, requires_grad=True)
    b0 = torch.zeros(1, dtype=torch.float64, requires_grad=True)
    Xt, yt = torch.tensor(X), torch.tensor(y)

    def sn(W):
        s = torch.linalg.matrix_norm(W, 2)
        return W * torch.clamp(bound / s, max=1.0)

    def feats(Z):
        h = Z @ W0
        for W, b in zip(Ws, bs):
            h = h + torch.relu(h @ sn(W) + b)
        return np.sqrt(2.0 / D) * torch.cos(h @ R + ph)

    params = Ws + bs + [beta, b0]
    opt = torch.optim.Adam(params, lr=lr)
    N = len(y)
    for _ in range(steps):
        opt.zero_grad()
        z = feats(Xt) @ beta + b0
        loss = (torch.nn.functional.softplus(z) - yt * z).sum() + 0.5 * (beta ** 2).sum()
        loss.backward(); opt.step()
    with torch.no_grad():
        Phi = feats(Xt); z = Phi @ beta + b0; p = torch.sigmoid(z)
        prec = torch.eye(D, dtype=torch.float64) + (Phi * (p * (1 - p))[:, None]).T @ Phi
        Sinv = torch.linalg.inv(prec)

    def predict(Xn):
        with torch.no_grad():
            Ph = feats(torch.tensor(Xn)); mu = Ph @ beta + b0
            var = ((Ph @ Sinv) * Ph).sum(1)
            return torch.sigmoid(mu / torch.sqrt(1 + np.pi * var / 8)).numpy()
    return predict


def moons_predictors(n=200, seed=6):
    """Predictive probabilities on a grid for the eight methods of Figure 17.15 (dict name -> (n*n,) array)."""
    from sklearn.gaussian_process import GaussianProcessClassifier
    from sklearn.gaussian_process.kernels import RBF, ConstantKernel
    torch = _torch()
    X, y, ood = moons_ood(n, seed=seed)
    gx, gy, G = _grid15()
    Gt = torch.tensor(G)
    net = FMLP([2, 8, 16, 16, 8, 1], "relu")
    P = {}
    t0 = time.time()
    th0 = lambda s: net.init(np.random.default_rng(s))
    th, _ = adam_fit(_bce_loss(net, X, y), th0(0), 1500, 0.01)
    with torch.no_grad():
        P["SGD"] = torch.sigmoid(net(torch.tensor(th), Gt)[:, 0]).numpy()
    # deep ensemble
    pe = []
    for s in range(10):
        th_, _ = adam_fit(_bce_loss(net, X, y), th0(10 + s), 1500, 0.01)
        with torch.no_grad():
            pe.append(torch.sigmoid(net(torch.tensor(th_), Gt)[:, 0]).numpy())
    P["Deep ensemble"] = np.mean(pe, 0)
    # MC dropout
    gen = torch.Generator().manual_seed(0)
    thd, _ = adam_fit(_bce_loss(net, X, y, drop=0.1, gen=gen), th0(0), 2000, 0.01)
    with torch.no_grad():
        P["MC dropout"] = np.mean([torch.sigmoid(net(torch.tensor(thd), Gt, drop=0.1, gen=gen)[:, 0]).numpy() for _ in range(50)], 0)
    # bootstrap
    pb = []
    rng = np.random.default_rng(1)
    for s in range(10):
        ib = rng.integers(0, len(y), len(y))
        th_, _ = adam_fit(_bce_loss(net, X[ib], y[ib]), th0(0), 1500, 0.01)
        with torch.no_grad():
            pb.append(torch.sigmoid(net(torch.tensor(th_), Gt)[:, 0]).numpy())
    P["Bootstrap"] = np.mean(pb, 0)
    print("   point-estimate methods %.0fs" % (time.time() - t0), flush=True)
    # HMC from the SGD solution
    rng = np.random.default_rng(3)
    S, acc = hmc_batched(bernoulli_mlp_logpost(net, X, y, 1.0), th, 500, 500, 25, 0.02, rng)
    with torch.no_grad():
        P["MCMC (HMC)"] = torch.sigmoid(net(torch.tensor(S[:, 0, :]), Gt)[..., 0]).mean(0).numpy()
    print("   HMC acceptance %.2f, %.0fs" % (acc.mean(), time.time() - t0), flush=True)
    # mean-field VI (Bayes by backprop)
    P["VI (mean field)"] = bbb_predict(net, X, y, Gt, th, seed=0)
    print("   VI done %.0fs" % (time.time() - t0), flush=True)
    gp = GaussianProcessClassifier(ConstantKernel(1.0) * RBF(1.0), random_state=0).fit(X, y.astype(int))
    P["GP classifier"] = gp.predict_proba(G)[:, 1]
    pred = sngp_fit(X, y, seed=0)
    P["SNGP"] = pred(G)
    print("   all done %.0fs" % (time.time() - t0), flush=True)
    return dict(P=P, X=X, y=y, ood=ood, gx=gx, gy=gy)


def bbb_predict(net, X, y, Gt, th_init, seed=0, steps=2500, n_mc=4, n_pred=100, lr=0.01, prior_sd=1.0):
    """Bayes by backprop (Section 17.3.3): q = N(mu, diag(sigma^2)), the negative ELBO estimated with the
    reparameterisation trick, mu started at th_init. Returns the Monte Carlo predictive probability on Gt."""
    torch = _torch()
    g = torch.Generator().manual_seed(seed)
    mu = torch.tensor(th_init, dtype=torch.float64, requires_grad=True)
    rho = torch.full_like(mu, -4.0).requires_grad_(True)
    Xt, yt = torch.tensor(X), torch.tensor(y)
    opt = torch.optim.Adam([mu, rho], lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        sig = torch.nn.functional.softplus(rho)
        eps = torch.randn(n_mc, mu.numel(), generator=g, dtype=torch.float64)
        th = mu + sig * eps
        z = net(th, Xt)[..., 0]
        nll = (torch.nn.functional.softplus(z) - yt * z).sum(-1).mean()
        kl = (torch.log(prior_sd / sig) + (sig ** 2 + mu ** 2) / (2 * prior_sd ** 2) - 0.5).sum()
        (nll + kl).backward(); opt.step()
    with torch.no_grad():
        sig = torch.nn.functional.softplus(rho)
        th = mu + sig * torch.randn(n_pred, mu.numel(), generator=g, dtype=torch.float64)
        return torch.sigmoid(net(th, Gt)[..., 0]).mean(0).numpy()


def fig_17_15(out=None):
    """Predictive probability on two moons and an out-of-distribution blob for eight methods."""
    cache = os.path.join(CACHE_DIR, "fig17_15.npz")
    if os.path.exists(cache):
        z = np.load(cache, allow_pickle=True); r = z["r"].item()
    else:
        r = moons_predictors()
        np.savez(cache, r=np.array(r, dtype=object))
    names = ["SGD", "Deep ensemble", "MC dropout", "Bootstrap", "MCMC (HMC)", "VI (mean field)", "GP classifier", "SNGP"]
    fig, axes = plt.subplots(2, 4, figsize=(12.5, 6.0), sharex=True, sharey=True)
    n = len(r["gx"])
    for k, (ax, nm) in enumerate(zip(axes.ravel(), names)):
        im = ax.contourf(r["gx"], r["gy"], r["P"][nm].reshape(n, n), levels=np.linspace(0, 1, 21), cmap="RdBu_r")
        ax.scatter(*r["X"][r["y"] == 0].T, c="C0", s=7, marker="o", edgecolor="w", lw=0.3)
        ax.scatter(*r["X"][r["y"] == 1].T, c="C3", s=7, marker="o", edgecolor="w", lw=0.3)
        ax.scatter(*r["ood"].T, c=GREEN, s=9, marker="^", edgecolor="k", lw=0.3)
        ax.set_title("(%s) %s" % ("abcdefgh"[k], nm), fontsize=9)
    fig.colorbar(im, ax=axes, fraction=0.02, label=r"$p(y=1\mid x)$")
    return savefig(fig, CH, "fig17-15", out)


# =========================================================================== 17.5.2 extended Kalman filter for an MLP
def ekf_target(u):
    """The test function (17.43) with the slip x -> u corrected: h*(u) = u - 10 cos(u) sin(u) + u^3."""
    return u - 10 * np.cos(u) * np.sin(u) + u ** 3


def mlp1_value_jac(th, u, H=6):
    """f(theta, u) = W2 tanh(W1 u + b1) + b2 for scalar u and its gradient w.r.t. theta (D = 3H + 1).
    Layout: W1 (H), b1 (H), W2 (H), b2 (1), the same order as FMLP([1, H, 1])."""
    W1, b1, W2, b2 = th[:H], th[H:2 * H], th[2 * H:3 * H], th[3 * H]
    h = np.tanh(W1 * u + b1)
    f = W2 @ h + b2
    d = W2 * (1 - h ** 2)
    return f, np.concatenate([d * u, d, h, [1.0]])


def ekf_mlp(us, ys, H=6, sigma=3.0, Sigma0=100.0, q=1e-4, theta0=None, groups=None, seed=0, record=()):
    """Extended Kalman filter for the weights of an MLP (Section 17.5.2): state theta, random-walk dynamics with
    covariance q I, observation y_t = f(theta, u_t) + N(0, sigma^2) linearised at the current mean. `groups` is a
    list of index arrays (decoupled EKF, Supplement 17.1.2): the covariance is block diagonal over the groups; None
    gives the global EKF. Returns the final (mean, covariance blocks) and the means and variances at the steps in
    `record`, and the running one-step-ahead squared errors."""
    D = 3 * H + 1
    rng = np.random.default_rng(seed)
    th = rng.normal(0, 0.5, D) if theta0 is None else theta0.copy()
    if groups is None:
        groups = [np.arange(D)]
    Sig = [Sigma0 * np.eye(len(g)) for g in groups]
    R = sigma ** 2
    snaps, errs = {}, []
    for t, (u, y) in enumerate(zip(us, ys), start=1):
        Sig = [S + q * np.eye(len(S)) for S in Sig]
        f, J = mlp1_value_jac(th, u, H)
        Jg = [J[g] for g in groups]
        Sk = R + sum(j @ S @ j for j, S in zip(Jg, Sig))
        K = [S @ j / Sk for j, S in zip(Jg, Sig)]
        xi = y - f
        errs.append(xi ** 2)
        for g, k in zip(groups, K):
            th[g] = th[g] + k * xi
        Sig = [S - np.outer(k, j @ S) for k, j, S in zip(K, Jg, Sig)]
        if t in record:
            snaps[t] = (th.copy(), [S.copy() for S in Sig])
    return th, Sig, snaps, np.array(errs)


def ekf_predict(th, Sig, groups, ug, H=6):
    """Mean and standard deviation of f(theta, u) under the linearised Gaussian posterior."""
    D = 3 * H + 1
    if groups is None:
        groups = [np.arange(D)]
    m, s = [], []
    for u in ug:
        f, J = mlp1_value_jac(th, u, H)
        m.append(f)
        s.append(np.sqrt(max(sum(J[g] @ S @ J[g] for g, S in zip(groups, Sig)), 0.0)))
    return np.array(m), np.array(s)


def node_groups(H=6):
    """Node-decoupled groups: the three weights of each hidden unit, and the output bias on its own."""
    return [np.array([j, H + j, 2 * H + j]) for j in range(H)] + [np.array([3 * H])]


def ekf_data(n=200, seed=0, sigma=3.0):
    rng = np.random.default_rng(seed)
    u = rng.uniform(-2, 2, n)
    return u, ekf_target(u) + sigma * rng.normal(size=n)


def fig_17_16(out=None):
    """Sequential Bayesian inference for an MLP with the extended Kalman filter, after 10, 20, 30 and 200 points."""
    u, y = ekf_data()
    steps = [10, 20, 30, 200]
    th, Sig, snaps, _ = ekf_mlp(u, y, record=steps, q=1e-3)
    ug = np.linspace(-2.5, 2.5, 200)
    fig, axes = plt.subplots(1, 4, figsize=(12.5, 2.9), sharey=True)
    for k, (ax, t) in enumerate(zip(axes, steps)):
        m, s = ekf_predict(snaps[t][0], snaps[t][1], None, ug)
        ax.plot(ug, ekf_target(ug), "k--", lw=0.9)
        ax.fill_between(ug, m - 2 * s, m + 2 * s, color=RED, alpha=0.25, lw=0)
        ax.plot(ug, m, color=RED, lw=1.5)
        ax.plot(u[:t], y[:t], ".", color=BLUE, ms=4)
        ax.set_title("step %d" % t, fontsize=9); ax.set_ylim(-25, 25); ax.set_xlabel("$u$")
        panel_label(ax, "abcd"[k], "lower right")
    fig.tight_layout()
    return savefig(fig, CH, "fig17-16", out)


def fig_17_S8(out=None):
    """Global, node-decoupled and fully decoupled EKF on the data of Figure 17.16."""
    u, y = ekf_data()
    H = 6
    full = [np.array([i]) for i in range(3 * H + 1)]
    runs = [("global EKF", None), ("node decoupled", node_groups(H)), ("fully decoupled (diagonal)", full)]
    fig, axes = plt.subplots(1, 2, figsize=(9.8, 3.3))
    ug = np.linspace(-2.5, 2.5, 200)
    for (nm, gr), col in zip(runs, [BLUE, GREEN, ORANGE]):
        th, Sig, _, err = ekf_mlp(u, y, groups=gr, q=1e-3)
        m, s = ekf_predict(th, Sig, gr, ug)
        axes[0].plot(ug, m, color=col, label=nm)
        axes[1].plot(np.arange(1, 201), np.cumsum(err) / np.arange(1, 201), color=col, label=nm)
        print("   %-28s final rmse to truth %.3f, mean one-step squared error %.2f" % (nm, np.sqrt(np.mean((m - ekf_target(ug)) ** 2)), err.mean()))
    axes[0].plot(ug, ekf_target(ug), "k--", lw=0.9, label="truth")
    axes[0].plot(u, y, ".", color=GREY, ms=2.5)
    axes[0].set_xlabel("$u$"); axes[0].set_ylim(-25, 25); axes[0].legend(fontsize=7.5)
    axes[1].set_xlabel("step $t$"); axes[1].set_ylabel("running mean of $(y_t-\\hat y_t)^2$"); axes[1].set_yscale("log")
    panel_label(axes[0], "a"); panel_label(axes[1], "b", "upper right")
    fig.tight_layout()
    return savefig(fig, CH, "fig17-S8", out)


# =========================================================================== 17.6 hierarchical Bayesian MLPs
def multimoons(J=18, n=100, noise=0.3, seed=31, spread=35.0):
    """J two-moons tasks, task j rotated by an angle in [-spread, spread] degrees; 50 training and 50 test points each."""
    ang = np.linspace(-spread, spread, J) * np.pi / 180
    Xtr, ytr, Xte, yte = [], [], [], []
    for j in range(J):
        X, y = make_moons_np(n, noise, seed + j)
        c, s = np.cos(ang[j]), np.sin(ang[j])
        X = (X - np.array([0.5, 0.25])) @ np.array([[c, s], [-s, c]])
        Xtr.append(X[:n // 2]); ytr.append(y[:n // 2]); Xte.append(X[n // 2:]); yte.append(y[n // 2:])
    return np.array(Xtr), np.array(ytr), np.array(Xte), np.array(yte), ang


def _task_logp_sep(net, Xtr, ytr):
    torch = _torch()
    Xt, yt = torch.tensor(Xtr), torch.tensor(ytr)

    def logp(th):                                  # th (J, P): one independent chain per task
        z = net(th, Xt)[..., 0]
        return (yt * z - torch.nn.functional.softplus(z)).sum(-1) - 0.5 * (th ** 2).sum(-1)
    return logp


def _hier_layer_index(net):
    idx = []
    for l, (a, b) in enumerate(net.shapes):
        idx += [l] * (a * b + b)
    return np.array(idx)


def _task_logp_hier(net, Xtr, ytr, J):
    """Non-centred hierarchical model (17.57): w_j = w0 + sigma_l eps_j, w0 ~ N(0,1), eps ~ N(0,1), sigma_l half-normal(1)
    through s_l = log sigma_l. The state is one vector (w0, s, eps_1, ..., eps_J)."""
    torch = _torch()
    P, Ls = net.P, len(net.shapes)
    lay = torch.tensor(_hier_layer_index(net))
    Xt, yt = torch.tensor(Xtr), torch.tensor(ytr)

    def split(th):
        w0 = th[..., :P]; s = th[..., P:P + Ls]; eps = th[..., P + Ls:].reshape(th.shape[:-1] + (J, P))
        return w0, s, eps

    def weights(th):
        w0, s, eps = split(th)
        sig = torch.exp(s)[..., lay]                # (..., P)
        return w0[..., None, :] + eps * sig[..., None, :]

    def logp(th):
        w0, s, eps = split(th)
        W = weights(th)                            # (B, J, P)
        z = net(W, Xt)[..., 0]                    # (B, J, N)
        ll = (yt * z - torch.nn.functional.softplus(z)).sum((-1, -2))
        lpr = -0.5 * (w0 ** 2).sum(-1) - 0.5 * (eps ** 2).sum((-1, -2)) + (-0.5 * torch.exp(2 * s) + s).sum(-1)
        return ll + lpr
    return logp, weights


def hier_moons(J=18, seed=31, n_warm=500, n_samp=300, L=20, widths=(2, 5, 5, 1)):
    """Separate and hierarchical MLPs for the multimoons problem; returns grids of mean predictive probability for
    the first four tasks and the train and test accuracy averaged over the tasks."""
    torch = _torch()
    Xtr, ytr, Xte, yte, ang = multimoons(J, seed=seed)
    net = FMLP(list(widths), "tanh")
    gx, gy, G = _moons_grid(lo=(-3.2, -2.6), hi=(3.2, 2.6), n=60)
    out = {}
    rng = np.random.default_rng(seed)
    # separate models: J independent chains
    th0 = np.stack([net.init(rng, gain=1.0) for _ in range(J)])
    logp = _task_logp_sep(net, Xtr, ytr)
    t0 = time.time()
    th_map, _ = adam_fit(lambda t: -logp(t).sum(), th0.ravel(), 1500, 0.02) if False else (None, None)
    Th = torch.tensor(th0, requires_grad=True)
    opt = torch.optim.Adam([Th], lr=0.02)
    for _ in range(1500):
        opt.zero_grad(); (-logp(Th).sum()).backward(); opt.step()
    S, acc = hmc_batched(logp, Th.detach().numpy(), n_warm, n_samp, L, 0.05, rng)       # (n_samp, J, P)
    print("   separate HMC accept %.2f, %.0fs" % (acc.mean(), time.time() - t0), flush=True)
    with torch.no_grad():
        St = torch.tensor(S)                                                   # (n_samp, J, P)
        def probs(X):
            Xb = torch.tensor(X)[:, None]
            return torch.sigmoid(net(St.permute(1, 0, 2), Xb)[..., 0]).mean(1).numpy()   # (J, N)
        ptr, pte = probs(Xtr), probs(Xte)
        pg = torch.sigmoid(net(St[:, :4, :].permute(1, 0, 2), torch.tensor(G))[..., 0]).mean(1).numpy()
    out["sep"] = dict(train=float(((ptr > .5) == ytr).mean()), test=float(((pte > .5) == yte).mean()), grid=pg)
    # hierarchical model: one chain
    logph, weights = _task_logp_hier(net, Xtr, ytr, J)
    P, Ls = net.P, len(net.shapes)
    w0 = rng.normal(0, 0.5, P); s0 = np.zeros(Ls); e0 = rng.normal(0, 0.3, J * P)
    Th = torch.tensor(np.concatenate([w0, s0, e0])[None, :], requires_grad=True)
    opt = torch.optim.Adam([Th], lr=0.01)
    for _ in range(2500):
        opt.zero_grad(); (-logph(Th).sum()).backward(); opt.step()
    S, acc = hmc_batched(logph, Th.detach().numpy(), n_warm * 2, n_samp, L, 0.02, rng)
    print("   hierarchical HMC accept %.2f, %.0fs" % (acc.mean(), time.time() - t0), flush=True)
    with torch.no_grad():
        W = weights(torch.tensor(S[:, 0, :]))                                      # (n_samp, J, P)
        def probs_h(X):
            return torch.sigmoid(net(W.permute(1, 0, 2), torch.tensor(X)[:, None])[..., 0]).mean(1).numpy()
        ptr, pte = probs_h(Xtr), probs_h(Xte)
        pg = torch.sigmoid(net(W[:, :4, :].permute(1, 0, 2), torch.tensor(G))[..., 0]).mean(1).numpy()
        sig = np.exp(S[:, 0, P:P + Ls])
    out["hier"] = dict(train=float(((ptr > .5) == ytr).mean()), test=float(((pte > .5) == yte).mean()), grid=pg, sigma=sig.mean(0))
    out.update(Xtr=Xtr, ytr=ytr, Xte=Xte, yte=yte, gx=gx, gy=gy, ang=ang)
    return out


def fig_17_17(out=None):
    """Two moons, and the multi-task version made by rotating the data (the first four tasks)."""
    Xtr, ytr, Xte, yte, ang = multimoons()
    X, y = make_moons_np(1000, 0.3, 31)
    fig = plt.figure(figsize=(10.5, 3.2))
    gs = fig.add_gridspec(1, 5, width_ratios=[1.3, 1, 1, 1, 1], wspace=0.3)
    ax = fig.add_subplot(gs[0])
    ax.scatter(*X[y == 0].T, c=BLUE, s=5); ax.scatter(*X[y == 1].T, c=RED, s=5)
    ax.set_title("two moons", fontsize=9); ax.set_xlabel("$x_1$"); ax.set_ylabel("$x_2$"); panel_label(ax, "a")
    for k in range(4):
        ax = fig.add_subplot(gs[1 + k])
        j = [0, 6, 12, 17][k]
        ax.scatter(*Xtr[j][ytr[j] == 0].T, c=BLUE, s=9); ax.scatter(*Xtr[j][ytr[j] == 1].T, c=RED, s=9)
        ax.set_title("task %d (%.0f deg)" % (j + 1, ang[j] * 180 / np.pi), fontsize=8.5); ax.set_xlim(-3, 3); ax.set_ylim(-2.5, 2.5)
        if k == 0: panel_label(ax, "b")
    return savefig(fig, CH, "fig17-17", out)


def fig_17_19(out=None):
    """Posterior predictive for four tasks: separate MLPs (top) and the hierarchical MLP (bottom)."""
    cache = os.path.join(CACHE_DIR, "fig17_19.npz")
    if os.path.exists(cache):
        r = np.load(cache, allow_pickle=True)["r"].item()
    else:
        r = hier_moons()
        np.savez(cache, r=np.array(r, dtype=object))
    n = len(r["gx"])
    fig, axes = plt.subplots(2, 4, figsize=(11.5, 5.4), sharex=True, sharey=True)
    for row, key, nm in [(0, "sep", "separate MLPs"), (1, "hier", "hierarchical MLP")]:
        for k in range(4):
            ax = axes[row, k]
            ax.contourf(r["gx"], r["gy"], r[key]["grid"][k].reshape(n, n), levels=np.linspace(0, 1, 11), cmap="RdBu_r", alpha=0.9)
            ax.scatter(*r["Xtr"][k][r["ytr"][k] == 0].T, c="C0", s=9, edgecolor="w", lw=0.3)
            ax.scatter(*r["Xtr"][k][r["ytr"][k] == 1].T, c="C3", s=9, edgecolor="w", lw=0.3)
            ax.set_title("%s, task %d" % (nm, k + 1), fontsize=8.5)
    print("   separate: train %.3f test %.3f; hierarchical: train %.3f test %.3f" % (r["sep"]["train"], r["sep"]["test"], r["hier"]["train"], r["hier"]["test"]))
    fig.tight_layout()
    return savefig(fig, CH, "fig17-19", out)


# =========================================================================== 17.3.10 the Laplace bridge
def laplace_bridge(m, V):
    """Dirichlet parameters from Gaussian logits N(m, V) by (17.29); m and V must already satisfy the zero-sum
    constraint (use project_zero_sum)."""
    m = np.asarray(m, float)
    C = len(m)
    return (1.0 / np.diag(V)) * (1 - 2.0 / C + np.exp(m) / C ** 2 * np.sum(np.exp(-m)))


def project_zero_sum(m, V):
    """Condition N(m, V) on 1'z = 0, equation (17.30)."""
    one = np.ones(len(m))
    Vo = V @ one
    s = one @ Vo
    return m - Vo * (one @ m) / s, V - np.outer(Vo, Vo) / s


def dirichlet_softmax_laplace(alpha):
    """Laplace approximation of a Dirichlet in the zero-sum softmax basis: mean log(alpha_k) - mean(log alpha),
    covariance P diag(1/alpha) P with the centring matrix P = I - 11'/C."""
    alpha = np.asarray(alpha, float)
    C = len(alpha)
    Pm = np.eye(C) - np.ones((C, C)) / C
    return np.log(alpha) - np.log(alpha).mean(), Pm @ np.diag(1.0 / alpha) @ Pm


def fig_17_S1(out=None):
    """The Laplace bridge. Top: Gaussian logits that come from the Laplace approximation of a Dirichlet (the bridge returns
    the same alpha); histogram of the first class probability of softmax(z) against Beta(alpha_1, alpha_0 - alpha_1).
    Bottom: logits N(m, s A) with a generic covariance."""
    rng = np.random.default_rng(0)
    xs = np.linspace(0.001, 0.999, 400)
    fig, axes = plt.subplots(2, 3, figsize=(10.5, 5.4))
    for ax, k in zip(axes[0], [4.0, 1.0, 0.25]):
        al = k * np.array([6.0, 3.0, 1.5])
        mu, V = dirichlet_softmax_laplace(al)
        z = mu + rng.normal(size=(40000, 3)) @ np.linalg.cholesky(V + 1e-12 * np.eye(3)).T
        pi = special.softmax(z, axis=1)
        a_b = laplace_bridge(mu, V)
        ax.hist(pi[:, 0], bins=60, density=True, color=GREY, alpha=0.55, label="softmax of Gaussian logits")
        ax.plot(xs, stats.beta.pdf(xs, a_b[0], a_b.sum() - a_b[0]), color=RED, lw=1.6, label="Beta from the bridge")
        ax.set_title(r"$\alpha=%g\cdot(6,3,1.5)$, $\alpha_0=%.1f$" % (k, a_b.sum()), fontsize=9); ax.set_xlabel(r"$\pi_1$")
        print("   row 1, k=%g: mean pi_1 MC %.3f, Beta %.3f" % (k, pi[:, 0].mean(), a_b[0] / a_b.sum()))
    m0 = np.array([1.2, 0.2, -1.4])
    A = np.array([[1.0, 0.3, -0.2], [0.3, 0.8, 0.1], [-0.2, 0.1, 1.2]])
    for ax, sc in zip(axes[1], [0.15, 0.6, 2.0]):
        V = sc * A
        mp, Vp = project_zero_sum(m0, V)
        al = laplace_bridge(mp, Vp)
        z = rng.multivariate_normal(m0, V, 40000)
        pi = special.softmax(z, axis=1)
        ax.hist(pi[:, 0], bins=60, density=True, color=GREY, alpha=0.55)
        ax.plot(xs, stats.beta.pdf(xs, al[0], al.sum() - al[0]), color=RED, lw=1.6)
        ax.set_title(r"generic covariance, scale %g, $\alpha_0=%.1f$" % (sc, al.sum()), fontsize=9); ax.set_xlabel(r"$\pi_1$")
        print("   row 2, s=%g: mean pi_1 MC %.3f, Beta %.3f" % (sc, pi[:, 0].mean(), al[0] / al.sum()))
    axes[0, 0].legend(fontsize=7.5)
    fig.tight_layout()
    return savefig(fig, CH, "fig17-S1", out)


# =========================================================================== 17.3.8 SGD as a Gaussian sampler, SWA
def sgd_quadratic(a, noise_sd, eps, n, seed=0, burn=2000):
    """Constant-step SGD on f(theta) = 0.5 sum a_i theta_i^2 with gradient noise N(0, diag(noise_sd^2)):
    theta <- theta - eps (a theta + xi). Returns the iterates after a burn-in."""
    rng = np.random.default_rng(seed)
    a = np.asarray(a, float); noise_sd = np.asarray(noise_sd, float)
    th = np.zeros(len(a)); out = []
    for t in range(n + burn):
        th = th - eps * (a * th + noise_sd * rng.normal(size=len(a)))
        if t >= burn:
            out.append(th.copy())
    return np.array(out)


def sgd_stationary_var(a, noise_sd, eps):
    """Exact stationary variances of the discrete recursion: eps^2 s^2 / (1 - (1 - eps a)^2)."""
    a = np.asarray(a, float)
    return eps ** 2 * np.asarray(noise_sd, float) ** 2 / (1 - (1 - eps * a) ** 2)


def fig_17_S2(out=None):
    """Constant-step SGD on a quadratic: the iterates, the stationary Gaussian and the SWA average."""
    a = np.array([8.0, 1.0]); s = np.array([2.0, 2.0]); eps = 0.05
    th = sgd_quadratic(a, s, eps, 20000, seed=1)
    v = sgd_stationary_var(a, s, eps)
    fig, axes = plt.subplots(1, 2, figsize=(9.6, 3.6))
    ax = axes[0]
    ax.plot(th[::10, 0], th[::10, 1], ".", color=GREY, ms=2, alpha=0.6)
    for k in (1, 2):
        t = np.linspace(0, 2 * np.pi, 200)
        ax.plot(k * np.sqrt(v[0]) * np.cos(t), k * np.sqrt(v[1]) * np.sin(t), color=RED, lw=1.0)
    swa = th[::100].mean(0)
    ax.plot(*swa, "*", color=BLUE, ms=13, label="SWA average of 200 iterates"); ax.plot(0, 0, "k+", ms=9, label="minimum")
    ax.set_xlabel(r"$\theta_1$"); ax.set_ylabel(r"$\theta_2$"); ax.legend(fontsize=7.5, loc="upper right"); ax.set_aspect("equal", "datalim")
    ax.set_title("iterates, 1 and 2 sd ellipses of the stationary law", fontsize=8.5); panel_label(ax, "a", "lower left")
    ax = axes[1]
    D = 100
    aD = np.linspace(0.5, 4, D); sD = np.full(D, 1.0)
    thD = sgd_quadratic(aD, sD, 0.05, 30000, seed=2)
    Ks = np.array([1, 2, 5, 10, 20, 50, 100, 200, 400])
    d2 = [np.mean([np.sum(thD[i * 50:(i + K) * 50:50].mean(0) ** 2) for i in range(0, max(1, 400 // K))]) for K in Ks]
    ax.loglog(Ks, d2, "o-", color=BLUE, label="SWA average of $K$ iterates (every 50 steps)")
    ax.axhline(np.sum(sgd_stationary_var(aD, sD, 0.05)), color=RED, ls="--", label=r"single iterate, tr$\,\Sigma$")
    ax.set_xlabel("$K$"); ax.set_ylabel(r"squared distance to the minimum"); ax.legend(fontsize=7.5)
    ax.set_title("$D=100$: averaging moves the iterate to the centre", fontsize=8.5); panel_label(ax, "b", "lower left")
    fig.tight_layout()
    return savefig(fig, CH, "fig17-S2", out)


# =========================================================================== 17.3.9.2 randomize-then-optimise
def bayes_linreg_posterior(Phi, y, sigma, tau):
    prec = Phi.T @ Phi / sigma ** 2 + np.eye(Phi.shape[1]) / tau ** 2
    S = np.linalg.inv(prec)
    return S @ Phi.T @ y / sigma ** 2, S


def rto_samples(Phi, y, sigma, tau, M, rng, noise=True, prior=True):
    """Randomize-then-optimise: theta_hat = argmin |y + eps - Phi theta|^2 / sigma^2 + |theta - theta0|^2 / tau^2 with
    eps ~ N(0, sigma^2 I) and theta0 ~ N(0, tau^2 I). Exact posterior draws when both perturbations are used."""
    N, D = Phi.shape
    A = Phi.T @ Phi / sigma ** 2 + np.eye(D) / tau ** 2
    out = []
    for _ in range(M):
        th0 = tau * rng.normal(size=D) if prior else np.zeros(D)
        eps = sigma * rng.normal(size=N) if noise else np.zeros(N)
        out.append(np.linalg.solve(A, Phi.T @ (y + eps) / sigma ** 2 + th0 / tau ** 2))
    return np.array(out)


def fig_17_S3(out=None):
    """Random-prior (randomize-then-optimise) ensembles for a linear model against the exact posterior."""
    rng = np.random.default_rng(0)
    x = rng.uniform(-0.2, 0.6, 12)
    feats = lambda x: np.cos(np.pi * np.outer(x, np.arange(8)))
    sigma, tau = 0.1, 1.5
    y = 0.5 + 0.4 * np.sin(5 * x) + sigma * rng.normal(size=x.size)
    Phi = feats(x); xg = np.linspace(-1, 1.5, 200); Pg = feats(xg)
    mu, S = bayes_linreg_posterior(Phi, y, sigma, tau)
    sd_ex = np.sqrt(np.einsum("ij,jk,ik->i", Pg, S, Pg))
    fig, axes = plt.subplots(1, 3, figsize=(11.2, 3.1), sharey=True)
    for ax, (nm, kw) in zip(axes, [("noise and prior perturbed (exact)", dict()), ("prior perturbed only", dict(noise=False)), ("noise perturbed only", dict(prior=False))]):
        T = rto_samples(Phi, y, sigma, tau, 4000, rng, **kw)
        F = T @ Pg.T
        ax.fill_between(xg, mu @ Pg.T - 2 * sd_ex, mu @ Pg.T + 2 * sd_ex, color=RED, alpha=0.2, label="exact posterior $\\pm2$ sd")
        ax.plot(xg, F.mean(0) + 2 * F.std(0), color=BLUE, lw=1.1, label="ensemble $\\pm2$ sd"); ax.plot(xg, F.mean(0) - 2 * F.std(0), color=BLUE, lw=1.1)
        ax.plot(x, y, "k.", ms=5); ax.set_title(nm, fontsize=8.5); ax.set_xlabel("$x$"); ax.set_ylim(-1.5, 2.5)
    axes[0].legend(fontsize=7.5, loc="upper left")
    fig.tight_layout()
    return savefig(fig, CH, "fig17-S3", out)


# =========================================================================== 17.3.1 MC dropout does not shrink
def dropout_linear_sd(N, p, D=5, sigma=0.5, tau=1.0, seed=0, n_mc=4000):
    """At a fixed test input, the predictive sd of (a) the exact Bayesian posterior, (b) MC dropout on the inputs of a
    linear model fitted by dropout-regularised least squares (inverted dropout, rate p), (c) bootstrap OLS ensembles."""
    rng = np.random.default_rng(seed)
    w = np.linspace(1.0, -0.5, D)
    X = rng.normal(size=(N, D)); y = X @ w + sigma * rng.normal(size=N)
    x0 = np.ones(D)
    mu, S = bayes_linreg_posterior(X, y, sigma, tau)
    bayes = np.sqrt(x0 @ S @ x0)
    # dropout least squares: minimise E_mask |y - X (m * w)/(1-p)|^2 = |y - Xw|^2 + p/(1-p) w' diag(X'X) w
    A = X.T @ X + p / (1 - p) * np.diag(np.diag(X.T @ X))
    w_hat = np.linalg.solve(A, X.T @ y)
    mk = (rng.uniform(size=(n_mc, D)) > p) / (1 - p)
    mcd = np.std((mk * w_hat) @ x0)
    bs = []
    for _ in range(300):
        i = rng.integers(0, N, N)
        bs.append(np.linalg.lstsq(X[i], y[i], rcond=None)[0] @ x0)
    return bayes, mcd, np.std(bs), float(np.sqrt(p / (1 - p) * np.sum(w_hat ** 2 * x0 ** 2)))


def fig_17_S4(out=None):
    """Predictive standard deviation at a fixed input against the training-set size N."""
    Ns = np.array([20, 50, 100, 300, 1000, 3000, 10000])
    R = np.array([dropout_linear_sd(n, 0.2) for n in Ns])
    fig, ax = plt.subplots(figsize=(5.2, 3.4))
    ax.loglog(Ns, R[:, 0], "o-", color=RED, label="exact Bayes")
    ax.loglog(Ns, R[:, 1], "s-", color=BLUE, label="MC dropout ($p=0.2$)")
    ax.loglog(Ns, R[:, 3], "k:", lw=1.0, label=r"$\sqrt{p/(1-p)\sum_i \hat w_i^2x_i^2}$")
    ax.loglog(Ns, R[:, 2], "^-", color=GREEN, label="bootstrap OLS")
    ax.set_xlabel("training set size $N$"); ax.set_ylabel("predictive sd of $f(x_0)$"); ax.legend(fontsize=7.5)
    fig.tight_layout()
    return savefig(fig, CH, "fig17-S4", out)


# =========================================================================== 17.4.6.1 BMA against MAP under a shift
def shift_data(n=40, seed=7):
    """One informative input x1 with a band boundary (class 1 iff |x1| < 1, 5% label noise) and a second input that
    is exactly zero in the training set (a dead input)."""
    rng = np.random.default_rng(seed)
    x1 = rng.uniform(-3, 3, n)
    y = (np.abs(x1) < 1).astype(float)
    flip = rng.uniform(size=n) < 0.05
    y[flip] = 1 - y[flip]
    return x1, y


def shift_logpost_fn(x1, y, H=4):
    """Log posterior over the 3H+1 live parameters (w, b, v, c) of f = v' tanh(w x1 + b) + c, N(0,1) priors, logistic
    likelihood. The weights from the dead input have no effect on the likelihood, so their posterior is their prior."""
    torch = _torch()
    xt, yt = torch.tensor(x1), torch.tensor(y)

    def logp(th):
        w, b, v, c = th[:, :H], th[:, H:2 * H], th[:, 2 * H:3 * H], th[:, 3 * H]
        z = torch.tanh(xt[None, :, None] * w[:, None, :] + b[:, None, :]) @ v[:, :, None]
        z = z[..., 0] + c[:, None]
        return (yt * z - torch.nn.functional.softplus(z)).sum(1) - 0.5 * (th ** 2).sum(1)
    return logp


def shift_posterior(seed=7, H=4, n_keep=200):
    """MAP and HMC draws of the live parameters (13 for H = 4) on the dead-input problem."""
    torch = _torch()
    x1, y = shift_data(seed=seed)
    logp = shift_logpost_fn(x1, y, H)
    rng = np.random.default_rng(seed)
    best = None
    for r in range(6):
        th0 = rng.normal(0, 1, 3 * H + 1)
        th, f = adam_fit(lambda t: -logp(t[None, :]).sum(), th0, 1500, 0.03)
        if best is None or f < best[1]:
            best = (th, f)
    S, acc = hmc_batched(logp, best[0][None, :], 500, 2000, 25, 0.1, rng, thin=10)
    return dict(x1=x1, y=y, map=best[0], samples=S[:, 0, :][:n_keep], acc=float(acc[0]))


def shift_predict(theta, x1, x2, dead, H=4):
    """p(y=1 | x1, x2) for parameter vectors theta (K, 3H+1) and dead weights `dead` (K, H) acting on x2."""
    w, b, v, c = theta[:, :H], theta[:, H:2 * H], theta[:, 2 * H:3 * H], theta[:, 3 * H]
    a = np.tanh(x1[None, :, None] * w[:, None, :] + b[:, None, :] + x2[None, :, None] * dead[:, None, :])
    return sigmoid(np.einsum("knh,kh->kn", a, v) + c[:, None])


def demo_shift_run(post, test_x1, test_y, z, s, tau, K=200, seed=1):
    """What the shift demonstration computes: accuracy and mean log probability of the true label for MAP (dead
    weights 0) and for the BMA with dead-input prior sd tau (the dead weights are fresh draws from N(0, tau^2) paired with
    the posterior draws), when the second input of the test points is s * z."""
    r = DKRng(seed)
    dead = np.array([[tau * r.norm() for _ in range(4)] for _ in range(K)])
    x2 = s * z
    pm = shift_predict(post["map"][None, :], test_x1, x2, np.zeros((1, 4)))[0]
    pb = shift_predict(post["samples"][:K], test_x1, x2, dead).mean(0)
    f = lambda p: (float(((p > 0.5) == (test_y > 0.5)).mean()), float(np.mean(np.log(np.where(test_y > 0.5, p, 1 - p)))))
    return dict(map=f(pm), bma=f(pb))


def shift_test_z(n=400, seed=5):
    r = DKRng(seed)
    return np.array([r.norm() for _ in range(n)])


def fig_17_S5(out=None):
    """BMA with an isotropic prior against MAP and against an empirical-covariance-style prior, as the corruption grows."""
    post = shift_posterior()
    rng = np.random.default_rng(11)
    tx = rng.uniform(-3, 3, 400); ty = (np.abs(tx) < 1).astype(float); z = shift_test_z()
    ss = np.linspace(0, 6, 25)
    res = {nm: [demo_shift_run(post, tx, ty, z, s, tau) for s in ss] for nm, tau in [("iso", 1.0), ("emp", 0.05)]}
    fig, axes = plt.subplots(1, 2, figsize=(9.4, 3.3))
    for k, lab in enumerate(["accuracy", "mean log probability of the true label"]):
        ax = axes[k]
        ax.plot(ss, [r["map"][k] for r in res["iso"]], color=GREEN, label="MAP")
        ax.plot(ss, [r["bma"][k] for r in res["iso"]], color=RED, label=r"BMA, prior sd 1 on the dead input")
        ax.plot(ss, [r["bma"][k] for r in res["emp"]], color=BLUE, label=r"BMA, prior sd 0.05")
        ax.set_xlabel("corruption size $s$"); ax.set_ylabel(lab); panel_label(ax, "ab"[k], "lower left")
    axes[0].legend(fontsize=7.5, loc="lower left", bbox_to_anchor=(0.02, 0.12))
    fig.tight_layout()
    print("   shift posterior: HMC acceptance %.2f; at s=4: " % post["acc"], res["iso"][16])
    return savefig(fig, CH, "fig17-S5", out)


# =========================================================================== 17.4.3 effective dimensionality
def effective_dimension(eigs, c):
    """N_eff(H, c) = sum_i lambda_i / (lambda_i + c), equation (17.36)."""
    e = np.clip(np.asarray(eigs, float), 0, None)
    return float(np.sum(e / (e + c)))


def width_study(widths=(2, 4, 8, 16, 32, 64), n=60, sigma=0.1, seed=0, steps=4000):
    """One-hidden-layer tanh MLPs of increasing width fitted to a small 2-d regression problem: train and test error,
    the spectrum of the Hessian of the negative log likelihood and N_eff with c = 1."""
    torch = _torch()
    rng = np.random.default_rng(seed)
    f = lambda X: np.sin(3 * X[:, 0]) + X[:, 1] ** 2
    X = rng.uniform(-1, 1, (n, 2)); y = f(X) + sigma * rng.normal(size=n)
    Xte = rng.uniform(-1, 1, (1000, 2)); yte = f(Xte)
    Xt, yt = torch.tensor(X), torch.tensor(y)
    res = []
    for h in widths:
        net = FMLP([2, h, 1], "tanh")
        nll = lambda t: ((net(t, Xt)[:, 0] - yt) ** 2).sum() / (2 * sigma ** 2)
        th, _ = adam_fit(lambda t: nll(t) + 0.5 * 1e-3 * (t ** 2).sum(), net.init(np.random.default_rng(seed + h)), steps, 0.01)
        Hm = torch.autograd.functional.hessian(nll, torch.tensor(th)).numpy()
        ev = np.linalg.eigvalsh((Hm + Hm.T) / 2)
        with torch.no_grad():
            tr = float(((net(torch.tensor(th), Xt)[:, 0] - yt) ** 2).mean()); te = float(((net(torch.tensor(th), torch.tensor(Xte))[:, 0] - torch.tensor(yte)) ** 2).mean())
        res.append(dict(h=h, P=net.P, eig=ev, neff=effective_dimension(ev, 1.0), train=tr, test=te))
    return res


def fig_17_S6(out=None):
    """Hessian spectrum, effective dimensionality and test error of MLPs of increasing width."""
    R = width_study()
    fig, axes = plt.subplots(1, 3, figsize=(11.5, 3.2))
    cm = plt.cm.viridis(np.linspace(0, 0.9, len(R)))
    for r, col in zip(R, cm):
        ev = np.sort(np.clip(r["eig"], 1e-12, None))[::-1]
        axes[0].semilogy(np.arange(1, len(ev) + 1), ev, color=col, label="width %d" % r["h"])
    axes[0].axhline(1.0, color="k", ls=":", lw=0.8); axes[0].set_xlabel("eigenvalue index"); axes[0].set_ylabel("Hessian eigenvalue"); axes[0].legend(fontsize=6.5, ncol=2)
    P = [r["P"] for r in R]
    axes[1].loglog(P, P, "k:", label="number of parameters"); axes[1].loglog(P, [r["neff"] for r in R], "o-", color=BLUE, label=r"$N_{\rm eff}$ ($c=1$)")
    axes[1].set_xlabel("number of parameters $P$"); axes[1].legend(fontsize=7.5)
    axes[2].semilogy(P, [r["train"] for r in R], "s-", color=GREEN, label="train error"); axes[2].semilogy(P, [r["test"] for r in R], "o-", color=RED, label="test error")
    axes[2].set_xscale("log"); axes[2].set_xlabel("number of parameters $P$"); axes[2].legend(fontsize=7.5)
    for k, ax in enumerate(axes):
        panel_label(ax, "abc"[k], "upper right" if k == 0 else "upper left")
    fig.tight_layout()
    for r in R:
        print("   width %2d  P=%3d  N_eff=%.1f  train %.4f  test %.4f" % (r["h"], r["P"], r["neff"], r["train"], r["test"]))
    return savefig(fig, CH, "fig17-S6", out)


# =========================================================================== 17.4.4 evidence of structured and shuffled labels
def gp_log_evidence(x, y, ell, sigma, amp=1.0):
    d = x[:, None] - x[None, :]
    K = amp ** 2 * np.exp(-0.5 * d ** 2 / ell ** 2) + sigma ** 2 * np.eye(len(x))
    return float(stats.multivariate_normal(np.zeros(len(x)), K).logpdf(y))


def fig_17_S7(out=None):
    """Log evidence of the true labels and of 300 shuffled copies under GP models with three length scales."""
    rng = np.random.default_rng(0)
    x = np.sort(rng.uniform(-3, 3, 50)); sigma = 0.1
    y = np.sin(1.3 * x) + sigma * rng.normal(size=50)
    ells = [0.03, 0.4, 1.5]
    fig, axes = plt.subplots(1, 3, figsize=(10.8, 3.0), sharex=True)
    for ax, ell in zip(axes, ells):
        sh = np.array([gp_log_evidence(x, rng.permutation(y), ell, sigma) for _ in range(300)])
        ev = gp_log_evidence(x, y, ell, sigma)
        ax.hist(sh, bins=25, color=GREY, alpha=0.7, label="shuffled labels")
        ax.axvline(ev, color=RED, lw=2, label="true labels")
        ax.set_title(r"length scale $%.2g$" % ell, fontsize=9); ax.set_xlabel("log evidence")
        print("   ell %.2f: true %.1f  shuffled mean %.1f sd %.1f" % (ell, ev, sh.mean(), sh.std()))
    axes[0].legend(fontsize=7.5)
    fig.tight_layout()
    return savefig(fig, CH, "fig17-S7", out)


# =========================================================================== the regression demonstration (Section 17.3)
REG_H, REG_SIG, REG_PRIOR = 16, 0.15, 1.0


def reg_data(seed=3):
    """The 1-d training set of the posterior-comparison demonstration: 12 points in [-2.6, -1] and 12 in [1, 2.6]."""
    rng = np.random.default_rng(seed)
    x = np.r_[np.sort(rng.uniform(-2.6, -1.0, 12)), np.sort(rng.uniform(1.0, 2.6, 12))]
    y = np.sin(1.5 * x) + 0.3 * x + REG_SIG * rng.normal(size=x.size)
    return np.round(x, 4), np.round(y, 4)


def reg_init(seed, H=REG_H):
    r = DKRng(seed)
    w = np.array([r.norm() for _ in range(H)]); b = np.array([0.5 * r.norm() for _ in range(H)])
    v = np.array([r.norm() / np.sqrt(H) for _ in range(H)])
    return np.concatenate([w, b, v, [0.0]])


def reg_forward(th, x, mask=None, H=REG_H):
    w, b, v, c = th[:H], th[H:2 * H], th[2 * H:3 * H], th[3 * H]
    h = np.tanh(np.outer(x, w) + b)
    hm = h if mask is None else h * mask
    return hm @ v + c, h


def reg_grad(th, x, y, mask=None, sigma=REG_SIG, prior=REG_PRIOR, H=REG_H):
    w, b, v, c = th[:H], th[H:2 * H], th[2 * H:3 * H], th[3 * H]
    f, h = reg_forward(th, x, mask, H)
    r = (f - y) / sigma ** 2
    m = np.ones(H) if mask is None else mask
    d = (1 - h ** 2) * (v * m)
    g = np.concatenate([(r[:, None] * d * x[:, None]).sum(0), (r[:, None] * d).sum(0), (r[:, None] * h * m).sum(0), [r.sum()]])
    return g + th / prior ** 2


def adam_np(grad_fn, th, steps, lr=0.01, b1=0.9, b2=0.999, eps=1e-8):
    th = th.copy(); m = np.zeros_like(th); v = np.zeros_like(th)
    for t in range(1, steps + 1):
        g = grad_fn(th, t)
        m = b1 * m + (1 - b1) * g; v = b2 * v + (1 - b2) * g * g
        th = th - lr * (m / (1 - b1 ** t)) / (np.sqrt(v / (1 - b2 ** t)) + eps)
    return th


def reg_map(x, y, seed, steps=3000, drop=0.0, drop_seed=0, H=REG_H):
    """MAP estimate by Adam from the demo-kit initialisation; with drop > 0 each step uses a fresh dropout mask on the
    hidden units (inverted dropout) drawn from the demo kit's generator."""
    r = DKRng(drop_seed)

    def gf(th, t):
        mask = None
        if drop > 0:
            mask = np.array([(1.0 if r.u() > drop else 0.0) / (1 - drop) for _ in range(H)])
        return reg_grad(th, x, y, mask, H=H)
    return adam_np(gf, reg_init(seed, H), steps)


def reg_jac(th, xg, H=REG_H):
    w, b, v = th[:H], th[H:2 * H], th[2 * H:3 * H]
    h = np.tanh(np.outer(xg, w) + b)
    d = (1 - h ** 2) * v
    return np.hstack([d * xg[:, None], d, h, np.ones((len(xg), 1))]), h


def reg_laplace(th, x, y, xg, sigma=REG_SIG, prior=REG_PRIOR, H=REG_H, last_layer=False):
    """Linearised (GGN) Laplace predictive sd of f on xg: sqrt(J(x) Sigma J(x)') with Sigma^{-1} = J'J/sigma^2 + I/prior^2;
    the last-layer version only treats (v, c) as random."""
    J, h = reg_jac(th, x, H)
    Jg, hg = reg_jac(th, xg, H)
    if last_layer:
        F = np.hstack([h, np.ones((len(x), 1))]); Fg = np.hstack([hg, np.ones((len(xg), 1))])
        S = np.linalg.inv(F.T @ F / sigma ** 2 + np.eye(H + 1) / prior ** 2)
        return np.sqrt(np.einsum("ij,jk,ik->i", Fg, S, Fg))
    S = np.linalg.inv(J.T @ J / sigma ** 2 + np.eye(J.shape[1]) / prior ** 2)
    return np.sqrt(np.einsum("ij,jk,ik->i", Jg, S, Jg))


def reg_ensemble(x, y, xg, M, seed0=100, steps=3000):
    F = [reg_forward(reg_map(x, y, seed0 + k, steps), xg)[0] for k in range(M)]
    return np.mean(F, 0), np.std(F, 0)


def reg_mcd(x, y, xg, p, seed=100, steps=3000, S=100, pred_seed=77, H=REG_H):
    th = reg_map(x, y, seed, steps, drop=p, drop_seed=seed + 1000)
    r = DKRng(pred_seed)
    F = []
    for _ in range(S):
        mask = np.array([(1.0 if r.u() > p else 0.0) / (1 - p) for _ in range(H)])
        F.append(reg_forward(th, xg, mask)[0])
    return np.mean(F, 0), np.std(F, 0)


def reg_hmc_reference(n_keep=1000):
    """Posterior of f(x) on the demo grid by HMC (49 parameters, N(0,1) priors, noise sd 0.15), started at the MAP."""
    torch = _torch()
    x, y = reg_data()
    xt, yt = torch.tensor(x), torch.tensor(y)
    H = REG_H

    def logp(th):
        w, b, v, c = th[:, :H], th[:, H:2 * H], th[:, 2 * H:3 * H], th[:, 3 * H]
        f = (torch.tanh(xt[None, :, None] * w[:, None, :] + b[:, None, :]) @ v[:, :, None])[..., 0] + c[:, None]
        return -((f - yt) ** 2).sum(1) / (2 * REG_SIG ** 2) - 0.5 * (th ** 2).sum(1) / REG_PRIOR ** 2
    th0 = reg_map(x, y, 100)
    rng = np.random.default_rng(0)
    S, acc = hmc_batched(logp, th0[None, :], 1000, 2 * n_keep, 30, 0.01, rng, thin=2)
    xg = reg_grid()
    F = np.array([reg_forward(s, xg)[0] for s in S[:, 0, :]])
    return dict(mean=F.mean(0), sd=F.std(0), q025=np.quantile(F, 0.025, 0), q975=np.quantile(F, 0.975, 0), acc=float(acc[0]))


def reg_grid():
    return np.linspace(-5, 5, 101)


def demo_post_run(M=5, p=0.1, steps=3000):
    """Everything the posterior-comparison demonstration computes for the default data."""
    x, y = reg_data(); xg = reg_grid()
    th = reg_map(x, y, 100, steps)
    out = dict(map=reg_forward(th, xg)[0])
    out["ens_mean"], out["ens_sd"] = reg_ensemble(x, y, xg, M, 100, steps)
    out["mcd_mean"], out["mcd_sd"] = reg_mcd(x, y, xg, p, 100, steps)
    out["lap_sd"] = reg_laplace(th, x, y, xg)
    out["lap_last_sd"] = reg_laplace(th, x, y, xg, last_layer=True)
    return out


# =========================================================================== exports, driver
def export_json():
    os.makedirs(DATA_OUT, exist_ok=True)
    x, y = reg_data()
    ref = reg_hmc_reference()
    d = dict(x=x.tolist(), y=y.tolist(), grid=np.round(reg_grid(), 4).tolist(), sigma=REG_SIG, prior_sd=REG_PRIOR, hidden=REG_H,
             hmc_mean=np.round(ref["mean"], 5).tolist(), hmc_sd=np.round(ref["sd"], 5).tolist(),
             hmc_lo=np.round(ref["q025"], 5).tolist(), hmc_hi=np.round(ref["q975"], 5).tolist(), hmc_accept=round(ref["acc"], 3))
    with open(os.path.join(DATA_OUT, "ch17-regression.json"), "w") as f:
        json.dump(d, f, separators=(",", ":"))
    print("wrote ../data/ch17-regression.json (HMC acceptance %.2f)" % ref["acc"])
    post = shift_posterior()
    rng = np.random.default_rng(11)
    tx = rng.uniform(-3, 3, 400); ty = (np.abs(tx) < 1).astype(float)
    d = dict(train_x=np.round(post["x1"], 5).tolist(), train_y=post["y"].tolist(), map=np.round(post["map"], 6).tolist(),
             samples=np.round(post["samples"], 6).tolist(), test_x=np.round(tx, 5).tolist(), test_y=ty.tolist(),
             z_seed=5, hidden=4, accept=round(post["acc"], 3))
    with open(os.path.join(DATA_OUT, "ch17-shift.json"), "w") as f:
        json.dump(d, f, separators=(",", ":"))
    print("wrote ../data/ch17-shift.json")


FIGURES = {"17.1": fig_17_1, "17.2": fig_17_2, "17.4": fig_17_4, "17.7": fig_17_7, "17.11": fig_17_11, "17.15": fig_17_15,
           "17.16": fig_17_16, "17.17": fig_17_17, "17.19": fig_17_19,
           "17.S1": fig_17_S1, "17.S2": fig_17_S2, "17.S3": fig_17_S3, "17.S4": fig_17_S4, "17.S5": fig_17_S5,
           "17.S6": fig_17_S6, "17.S7": fig_17_S7, "17.S8": fig_17_S8}


if __name__ == "__main__":
    if "--json" in sys.argv:
        export_json()
    else:
        common.run(FIGURES, CH)
        if "--fig" not in sys.argv and "--list" not in sys.argv:
            export_json()
