Interactive demo · Probabilistic Machine Learning · Chapter 11

Shrinkage and sparsity: ridge, lasso and subset selection

We minimise ½‖y − Xw‖2 plus a penalty. On the left the design is orthonormal, so each coefficient is treated separately: ridge divides the least squares value by 1 + λ, the lasso soft-thresholds it, and best-subset selection hard-thresholds it. On the right the design has two correlated features: the blue ellipses are the contours of the squared error around the least squares point, and the solutions of the two penalised problems are shown together with their paths as λ grows. Watch the lasso path hit the axis and stay there.

Penalty strength λ

Feature coupling A12

Least squares point w1

Least squares point w2

Try these

Right panel: the quadratic is ½(w − wls)TA(w − wls) with A = XTX = [[2, A12], [A12, 1]] (feature correlation A12/√2); the ridge penalty is ½λ‖w‖22 and the lasso penalty is λ‖w‖1. Left panel: hard threshold at √(2λ), the rule for the penalty λ per non-zero coefficient.

In the lecture notes