Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 17

When does a Bayesian model average lose to the MAP?

The network sees an informative input x1 and a second input x2 that was exactly zero in every training example. The weights from x2 do not touch the likelihood, so their posterior is their prior. The MAP sets them to zero and ignores any later corruption of x2; a Bayesian average draws them from the prior, and the corruption then scrambles the hidden units. Shrinking the prior variance of the dead weights (the idea of an empirical-covariance prior) removes the damage.

Corruption size s

Prior sd of the dead-input weights τ (log scale)

Try these

Network 1–4–1 with tanh units and logistic output, f = vᵀ tanh(w x1 + b + d x2) + c. Training: 40 points of a band problem (class 1 iff |x1| < 1, 5% label noise) with x2 = 0. The 13 live parameters have N(0, 1) priors and are sampled by HMC (200 draws, computed offline); the 4 dead weights d are fresh N(0, τ²) draws paired with them. Test: 400 points with the correct labels and x2 = s × a fixed standard normal. MAP: dead weights 0.

In the lecture notes