We minimise ½‖y − Xw‖2 plus a penalty. On the left the design is orthonormal, so each coefficient is treated separately: ridge divides the least squares value by 1 + λ, the lasso soft-thresholds it, and best-subset selection hard-thresholds it. On the right the design has two correlated features: the blue ellipses are the contours of the squared error around the least squares point, and the solutions of the two penalised problems are shown together with their paths as λ grows. Watch the lasso path hit the axis and stay there.
Right panel: the quadratic is ½(w − wls)TA(w − wls) with A = XTX = [[2, A12], [A12, 1]] (feature correlation A12/√2); the ridge penalty is ½λ‖w‖22 and the lasso penalty is λ‖w‖1. Left panel: hard threshold at √(2λ), the rule for the penalty λ per non-zero coefficient.
code/ch11_linear_regression.py (soft_threshold, lasso_cd, Figures 11.7 to 11.9).