Interactive demo · Statistical Learning: Theory and Methods · Chapter 4

Logistic regression on the heart disease data: IRLS step by step and the L1 path

462 men from a South African heart disease study, 160 of them with coronary heart disease (chd). The log-likelihood of a logistic regression is concave, and Newton–Raphson turns into iteratively reweighted least squares: each step regresses the adjusted response z = Xβ + (y − p)/(p(1 − p)) on X with weights p(1 − p) (4.26)–(4.28). Step through it from β = 0, drop variables and compare deviances (Tables 4.2 and 4.3), or switch to the L1 penalty (4.31), whose solution path adds one variable at a time; an active variable has its generalised correlation |xjT(y − p)| exactly equal to λ (4.32).

What to show

Variables in the model (click to drop or add)

IRLS step k

Penalty log₁₀(λ/λmax)

Try these

IRLS starts at β = 0 and halves a step if the log-likelihood would fall. Standard errors are the square roots of the diagonal of (XTWX)−1 at the current step; at convergence they are those of the maximum likelihood fit. For the L1 path the inputs are standardised to unit variance and the intercept is not penalised; each λ is solved by repeated weighted lasso (coordinate descent on the IRLS quadratic approximation).

In the lecture notes