Log prostate-specific antigen is regressed on eight clinical measurements for 67 men (standardised), and the fit is tested on 30 others. Choose ridge, the lasso or the elastic net and slide the penalty λ. The top left shows every coefficient along the whole path, the top right the coefficients at the current λ against least squares, and the bottom the training and test error. The lasso path is piecewise linear: the vertical ticks mark the critical values of λ where a variable enters or leaves.
Objective: ½‖y − Xw‖2 + λ‖w‖1 (lasso), ½‖y − Xw‖2 + ½λ‖w‖22 (ridge), and the lasso objective plus ½μ‖w‖22 (elastic net). The intercept is not penalised. At λ = ||XTy||∞ = 58.44 the lasso solution is zero.
code/ch11_linear_regression.py (lasso_cd, lars_lasso, Figures 11.10 to 11.12).