One data set of the simulation of Figure 18.1: N = 100 samples, p standard Gaussian features with pairwise correlation 0.2, coefficients drawn from N(0, 1), and noise set so that the signal-to-noise ratio is 2. The ridge fit is computed through the N×N matrix XXT (the reduction X = RVT of Section 18.3.5); on request it is also computed through the p×p normal equations, to check that the two fits are the same and to compare their costs.
The intercept is not penalised (inputs and response are centred). The relative test error is exact for the data set shown: 1 + (β̂ − β)TΣ(β̂ − β)/σ2. The Bayes risk is the error of the best possible rule (the posterior mean, which is ridge itself at λ = σ2), averaged over β. Times are measured in your browser.
code/ch18_high_dimensional_problems.py (ridge_svd, ridge_direct, bayes_rel_error, demo_ridge).