Interactive demo · Statistical Learning: Theory and Methods · Chapter 3

Ridge and lasso on the prostate data: paths, df(λ) and shrinkage

The log PSA of 67 men is regressed on eight standardised clinical measures. Ridge regression (3.41) adds λ∑βj2 to the residual sum of squares; the lasso (3.52) adds λ∑|βj|. Move the slider and watch the coefficients: ridge shrinks every coefficient smoothly and shrinks most along the principal directions in which the inputs vary least (bars, left), while the lasso sets coefficients exactly to zero, and a coefficient is nonzero exactly when its inner product with the residual has reached λ (bars, right).

Method

Complexity

Try these

Ridge is computed from the eigendecomposition XTX = VD2VT of the centred training inputs, λ is found from df(λ) by bisection, and the whole lasso path is computed exactly by least angle regression with the lasso modification (Algorithm 3.2a). The intercept is ȳ − xTβ, never penalised. Test error: mean squared error on the 30 test cases.

In the lecture notes