Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 17

How good are the approximate posteriors of a neural network?

A 1–16–1 tanh network with N(0, 1) priors is fitted to 24 points with a gap between them. The teal band is the posterior of the function from a long HMC run (49 parameters, started at the MAP, so it explores one mode); the gold curve and band belong to the method you choose. A single MAP network has no uncertainty at all; an ensemble, MC dropout and the linearised Laplace approximation each recover part of the posterior, and none recovers all of it.

Method

Ensemble members M

Dropout rate p

HMC reference

Try these

Model f(x) = ∑j vj tanh(wj x + bj) + c, noise sd 0.15, N(0, 1) prior on all 49 parameters. Every network is trained by 3000 Adam steps (rate 0.01) from a seeded initialisation. MC dropout trains one network with dropout on the 16 hidden units and averages 100 random masks. Laplace: the generalised Gauss–Newton matrix JᵀJ / σ² + I at the MAP, predictive sd of the linearised network; the last-layer version treats only (v, c) as random. The bands show ±2 standard deviations of the function f(x), not of a new observation.

In the lecture notes