Interactive demo · Foundations of Deep Learning · Chapter 9

L1 versus L2 regularisation: paths and sparsity

A linear model y = wTx with eight inputs, only three of which matter (true weights 3, 1.5, 0, 0, 2, 0, 0, 0), is fitted by minimising ½∑(t − wTx)2 + (λ/2)∑|wj|q (9.19). With q = 2 (ridge, weight decay) every coefficient shrinks smoothly and none is ever exactly zero; with q = 1 (the lasso) coefficients hit zero one after another as λ grows, leaving a sparse model. The right panel shows why: the minimum of the error inside the region ∑|wj|q ≤ η (9.20) sits at a corner when the region has corners.

Path shown (left panel)

Regularisation λ (log scale)

Correlation between neighbouring inputs ρ

Training points N

Geometry: exponent q (right panel)

Geometry: size of the region η

Try these

Inputs are standardised on the training set and the targets centred, so no bias is needed. The lasso is solved by cyclic coordinate descent (soft thresholding at λ/2), ridge in closed form, (XTX + λI)−1XTt. Validation error is measured on 200 fresh points from the same model (noise standard deviation 2).

In the lecture notes