Interactive demo · Probability and Statistical Inference · Chapter 13

Logistic regression on the CORIS data: Newton's method, the Fisher information and backward AIC

462 men from the Coronary Risk-Factor Study, 160 of them with coronary heart disease, and nine risk factors. The maximum likelihood estimate of a logistic regression has no closed form; the book computes it by reweighted least squares, which is Newton's method for this likelihood. Panel (a) races Newton against two slower algorithms that also climb the likelihood at every step. The standard errors come from the inverse of the Fisher information XTWX; panel (b) checks one of them against a parametric bootstrap. Choose the covariates, or let AIC remove them one at a time.

Covariates in the model

Coefficient shown in (b)

Bootstrap samples R

Try these

Newton (reweighted least squares, p. 224 with Z in step 3): β ← (XTWX)−1XTWZ. Bounded curvature: β ← β + 4(XTX)−1XT(y − p), using p(1 − p) ≤ 1/4. Gradient ascent: step 4/λmax(XTX). All three start at 0; solid lines use standardised covariates, dashed lines the raw units. Standard errors: square roots of the diagonal of (XTWX)−1 at the estimate. Likelihood ratio test: twice the drop in log-likelihood when the covariate is removed, against χ21. AIC = −2 log-likelihood + 2 × (number of coefficients), as in R. Bootstrap: responses redrawn from the fitted probabilities (seeded), the model refitted by Newton from the estimate.

In the lecture notes