Twenty-two points in the plane are separated by a line through the origin, so the maximum-likelihood weights are infinite and only a prior makes the problem well posed. With the prior N(0, σ2I) the right panel shows the posterior over the two weights (heat map), its Laplace approximation (ellipses centred at the MAP estimate) and a few posterior draws. The left panel shows the predictive probability p(y = 1 | x, D) in the input plane, computed by plugging in the MAP weights, by averaging over posterior draws (10.110), or by the moderated probit formula σ(κ(v)m) (10.113). Click in the left panel to move the probe point and compare the four numbers.
Data: 22 simulated points, class 1 (red crosses) on one side of the line with normal (2, 1) through the origin, class 0 (blue dots) on the other, none closer than 0.9 to the line. The model has no bias: p(y = 1 | x, w) = σ(wTx). The Laplace covariance is the inverse of XTSX + I/σ2 at the MAP. The "exact" value in the readout integrates σ(a) against the Gaussian law N(m, v) of a = wTx numerically; Monte Carlo uses a seeded random generator, so it is repeatable.
code/ch10_logistic_regression.py (laplace_logreg, predictive_mc, predictive_probit, predictive_exact).