Fifty observations, 300 independent Gaussian inputs, and a true linear predictor with k nonzero Gaussian coefficients. For each simulated data set the page computes the whole lasso path and the whole ridge path and keeps the best test error each can reach (the penalty chosen with the truth, so each method is shown at its very best). The top panel plots the fraction of the error of a constant model that each method removes, against k. With few nonzero coefficients the lasso removes most of the error and ridge little; when all 300 are nonzero ridge wins, but neither does well. That asymmetry is the principle: use a procedure that does well in sparse problems, since no procedure does well in dense ones.
Regression: Y = s·xTβ + N(0,1) noise, with s set so that Var(Y|x)/Var(s xTβ) is the chosen ratio; explained = 1 − PE/Var(Y). Classification: Pr(Y=1|x) = σ(s xTβ), the ratio is E[p(1−p)]/Var(p), both methods fit the 0/1 response by least squares and classify at 1/2; explained = 1 − error/0.5. Test errors are exact population values (no test sample). Lasso: best point on the entire path; ridge: best of 200 penalties and the least-squares limit.
code/ch16_ensemble_learning.py (best_lasso, best_ridge, err_classification, bos_scale, demo_bos_one); the numbers shown here are those of the Python functions on the same simulated data.