Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 5

Bayesian neural networks: Laplace approximation, evidence and predictive distribution

With a Gaussian prior N(w | 0, α⁻¹I) on the weights and Gaussian noise of precision β, the posterior of a network's weights is approximated by a Gaussian at the MAP weights w_MAP with precision A = βH + αI, where H is the exact Hessian of the sum-of-squares error. This gives the posterior samples, the predictive variance 1/β + g^TA⁻¹g, and the evidence for α and β, which can be re-estimated from γ, the number of well-determined parameters.

Hidden units M

Data points N

Weight precision α (log scale)

Noise precision β (log scale)

Weight precision α (log scale)

Prediction

Evidence procedure

Try these

Regression: x uniform on (0, 1), t = sin(2πx) + Gaussian noise (sd 0.2, so the true β is 25), seeded. The weights are found by Adam (1500 steps) and then a damped Newton method on βE_D + (α/2)‖w‖² with the exact Hessian (5.92)–(5.95); sliders move α or β and refit from the current weights, so the network stays in the same local minimum. One “round” of the evidence procedure refits w, then sets α = γ/w^Tw (5.178) and 1/β = 2E_D/(N − γ) (5.180)–(5.182), with γ = Σλ_i/(α + λ_i) over the eigenvalues of βH (negative ones, possible far from a good fit, are set to 0). Classification: our version of the two-class synthetic data (a Gaussian class and a mixture of two Gaussians), a network with 8 hidden units and a logistic output, one α, no β.

In the lecture notes

Chapter 5 notes: Section 5.7 (Bayesian neural networks), 5.7.1 (the posterior and the Laplace approximation), 5.7.2 (hyperparameter optimisation, γ), 5.7.3 (Bayesian classification networks and the predictive (5.190)).
Python code: code/ch05_neural_networks.py (hessian_exact, laplace_log_evidence, evidence_step_regression, predictive_regression, evidence_classification, predictive_classification; Figures 5.22 and 5.23) and the checks in code/checks_ch05.py. The same data and weights are produced by data_sine, classification_data and init_weights with JSRandom, and the Adam and Newton steps agree number for number.