Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 4

Logistic regression: IRLS, probit, Laplace and the Bayesian predictive

Logistic regression models p(C₁ | x) = σ(w^Tx) directly. The log likelihood is concave, and Newton–Raphson turns into iterative reweighted least squares: w ← (Φ^TRΦ)⁻¹Φ^TRz. If the classes are separable the likelihood has no maximum and |w| grows without bound; a Gaussian prior (the Laplace approximation to the posterior is then a Gaussian N(w_MAP, S_N)) keeps it finite, and integrating over it with the probit approximation gives the predictive probability σ(κ(σ_a²)μ_a), which is less confident than the MAP plug-in far from the data.

View

Class separation

Spread (standard deviation)

Points per class

Prior precision α (L2 term)

Model

IRLS iteration

Steepness a in σ(az + b)

Offset b

Show

five posterior samples of the boundary

Scoreboard

Try these

In the lecture notes

Section 4.3 (probabilistic discriminative models): 4.3.2 logistic regression, its gradient (4.91) and the separable case, 4.3.3 IRLS (4.92) to (4.101), 4.3.5 probit regression.
Section 4.4 (the Laplace approximation, (4.125) to (4.135), Figure 4.14) and Section 4.5 (Bayesian logistic regression: the posterior (4.142) to (4.143), the predictive distribution (4.145) to (4.155)).
Python code: code/ch04_linear_classification.py (functions irls_logistic, probit_irls, laplace_1d, laplace_logistic, bayes_predictive; Figures 4.9, 4.13, 4.14) and the checks in code/checks_ch04.py.