Interactive demo · Probabilistic Machine Learning · Chapter 10

Fitting logistic regression: gradient, bound and Newton steps

The 150 irises are classified as Virginica or not from the petal width alone, with a bias: two parameters (b, w) and a convex loss, the surface of Figure 10.5. The left panel shows the contours of the mean negative log-likelihood and the paths of three algorithms from the same start: plain gradient descent with a fixed step (10.28 with all N examples), the bound-optimisation iteration that uses the fixed matrix XTX/4 in place of the Hessian (10.82), and Newton's method, which is iteratively reweighted least squares (10.36). The right panel shows how far the loss is above its minimum after each iteration. The valley is long and narrow, so the first-order method crawls and the second-order methods do not.

Iterations K

Gradient-descent step log10 η

Feature

Start

Try these

Data: the iris petal width (cm) of all 150 flowers, label 1 for Virginica. The loss is the mean of the cross entropies (10.8); every method works on the same function. The "standardised" feature is (width − mean)/standard deviation, with the bias re-fitted. Newton's method uses step halving if a full step would raise the loss; it never does here. Stochastic gradient descent is not shown because its path depends on the order of the examples; it behaves like gradient descent with a noisy step.

In the lecture notes