Forty points with one input x are modelled by a GLM with linear predictor η = w0 + w1x and the canonical link: Gaussian (identity), Bernoulli (logit, i.e. logistic regression) or Poisson (log). The negative log likelihood is convex in w for all three (12.34). The right panel draws its contours, with the path of the chosen algorithm. Iteratively reweighted least squares is Newton's method here: each step solves a weighted least squares problem and, near the minimum, doubles the number of correct digits. For the Gaussian family it lands on the answer in one step. Gradient descent only follows the gradient (12.31) and needs a step size that suits the curvature.
Data: fixed simulated sets (N = 40, inputs uniform on [−1.5, 1.5]); true parameters (0.5, 1.2) for the Gaussian, (0.3, 1.6) for the Bernoulli and (0.6, 0.9) for the Poisson model. The dispersion is 1. The gradient step multiplies the gradient of the mean negative log likelihood. The contour levels are 0.5, 2, 5, 12, 30, 70, 150 and 300 above the minimum.
code/ch12_generalized_linear_models.py (glm_fit, glm_grad, glm_hessian, gradient_descent).