462 men from a South African heart disease study, 160 of them with coronary heart disease (chd). The log-likelihood of a logistic regression is concave, and Newton–Raphson turns into iteratively reweighted least squares: each step regresses the adjusted response z = Xβ + (y − p)/(p(1 − p)) on X with weights p(1 − p) (4.26)–(4.28). Step through it from β = 0, drop variables and compare deviances (Tables 4.2 and 4.3), or switch to the L1 penalty (4.31), whose solution path adds one variable at a time; an active variable has its generalised correlation |xjT(y − p)| exactly equal to λ (4.32).
IRLS starts at β = 0 and halves a step if the log-likelihood would fall. Standard errors are the square roots of the diagonal of (XTWX)−1 at the current step; at convergence they are those of the maximum likelihood fit. For the L1 path the inputs are standardised to unit variance and the intercept is not penalised; each λ is solved by repeated weighted lasso (coordinate descent on the IRLS quadratic approximation).
code/ch04_linear_classification.py (irls, logistic_summary, backward_elimination, l1_logistic, l1_knots, Figure 4.13).