Why does the lasso set coefficients exactly to zero while ridge regression never does? Three views of the same fact. Top: when the predictors are orthonormal (X = I) ridge shrinks every estimate by the same factor, the lasso moves every estimate towards zero by λ/2 and stops at zero, and best subset keeps or kills. Middle: with two real predictors (Hits and CRBI of the Hitters data) the lasso's budget is a diamond whose corners lie on the axes, the ridge budget a disc. Bottom: all 19 Hitters predictors, the paths as λ changes, and the 10-fold cross-validation curve that chooses λ.
Two-predictor panel: Salary on the standardised Hits and CRBI; the budget is the stated fraction of the least squares ℓ1 norm (lasso) or ℓ2 norm (ridge), and the solution is found by bisection on the penalty. Bottom panels: glmnet's scale, so the lasso minimises RSS/(2n) + λ∑|βj| on standardised columns and ridge solves (ZTZ + nλ/sy I)β = ZTy; the lasso is computed by coordinate descent. Coefficients are on the standardised scale; the cross-validation curves use ten fixed folds of the 263 players.
code/ch06_model_selection_regularization.py (Figures 6.7 and 6.10, and the data of this page) and code/ch06_lab.py.