Interactive demo · Applied Statistical Learning in R and Python · Chapter 6

Ridge and lasso: thresholds, constraint regions and λ

Why does the lasso set coefficients exactly to zero while ridge regression never does? Three views of the same fact. Top: when the predictors are orthonormal (X = I) ridge shrinks every estimate by the same factor, the lasso moves every estimate towards zero by λ/2 and stops at zero, and best subset keeps or kills. Middle: with two real predictors (Hits and CRBI of the Hitters data) the lasso's budget is a diamond whose corners lie on the axes, the ridge budget a disc. Bottom: all 19 Hitters predictors, the paths as λ changes, and the 10-fold cross-validation curve that chooses λ.

X = I: penalty λ

X = I: least squares estimate y

Two predictors: budget s (fraction of least squares)

All 19 predictors: method

log10 λ (glmnet scale)

Try these

Two-predictor panel: Salary on the standardised Hits and CRBI; the budget is the stated fraction of the least squares ℓ1 norm (lasso) or ℓ2 norm (ridge), and the solution is found by bisection on the penalty. Bottom panels: glmnet's scale, so the lasso minimises RSS/(2n) + λ∑|βj| on standardised columns and ridge solves (ZTZ + nλ/sy I)β = ZTy; the lasso is computed by coordinate descent. Coefficients are on the standardised scale; the cross-validation curves use ten fixed folds of the 263 players.

In the lecture notes