Interactive demo · Probabilistic Machine Learning · Chapter 10

Bayesian logistic regression: Laplace and the probit predictive

Twenty-two points in the plane are separated by a line through the origin, so the maximum-likelihood weights are infinite and only a prior makes the problem well posed. With the prior N(0, σ2I) the right panel shows the posterior over the two weights (heat map), its Laplace approximation (ellipses centred at the MAP estimate) and a few posterior draws. The left panel shows the predictive probability p(y = 1 | x, D) in the input plane, computed by plugging in the MAP weights, by averaging over posterior draws (10.110), or by the moderated probit formula σ(κ(v)m) (10.113). Click in the left panel to move the probe point and compare the four numbers.

Prior variance log10 σ2

Predictive

Posterior draws S

Try these

Data: 22 simulated points, class 1 (red crosses) on one side of the line with normal (2, 1) through the origin, class 0 (blue dots) on the other, none closer than 0.9 to the line. The model has no bias: p(y = 1 | x, w) = σ(wTx). The Laplace covariance is the inverse of XTSX + I/σ2 at the MAP. The "exact" value in the readout integrates σ(a) against the Gaussian law N(m, v) of a = wTx numerically; Monte Carlo uses a seeded random generator, so it is repeatable.

In the lecture notes