Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 15

Bayesian logistic regression: Laplace posterior and predictive

Twenty points in the plane are separable by a line through the origin, so the maximum likelihood weights are infinite. A spherical N(0, σ2I) prior gives a finite MAP estimate, and the Laplace approximation turns the posterior into a Gaussian. Left: the predictive probability of class 1. Right: the log posterior of the weights with the Gaussian approximation.

Predictive

Prior variance σ² (log scale)

Labels

Test point x*, first coordinate

Test point, second coordinate

Try these

The model has no bias term: p(y = 1 | x, w) = σ(wTx). The MAP estimate comes from Newton's method; the posterior is N(wMAP, H−1) with H the Hessian of the negative log posterior. "Sampled" draws 20 weight vectors and shows their boundaries; "Monte Carlo" averages σ(wTx) over 2000 draws; "probit" uses σ(μ/√(1 + πs2/8)) with μ = wMAPTx, s2 = xTH−1x. The "2 flipped" option changes the labels of the two points closest to the separating line.

In the lecture notes