Interactive demo · Statistical Learning: Theory and Methods · Chapter 18

Ridge regression when p ≫ N

One data set of the simulation of Figure 18.1: N = 100 samples, p standard Gaussian features with pairwise correlation 0.2, coefficients drawn from N(0, 1), and noise set so that the signal-to-noise ratio is 2. The ridge fit is computed through the N×N matrix XXT (the reduction X = RVT of Section 18.3.5); on request it is also computed through the p×p normal equations, to check that the two fits are the same and to compare their costs.

Features p

Penalty log₁₀ λ

Data set

Try these

The intercept is not penalised (inputs and response are centred). The relative test error is exact for the data set shown: 1 + (β̂ − β)TΣ(β̂ − β)/σ2. The Bayes risk is the error of the best possible rule (the posterior mean, which is ridge itself at λ = σ2), averaged over β. Times are measured in your browser.

In the lecture notes