Interactive demo · Statistical Learning: Theory and Methods · Chapter 3

PCR, PLS and ridge: shrinking along principal components

Write any linear fit in the coordinates of the principal components of the inputs. Least squares uses each component with factor 1; ridge regression multiplies component j by dj2/(dj2 + λ); principal components regression keeps the first M components and discards the rest; partial least squares, which also looks at the response, applies factors that are mostly below 1 for the low-variance components but can be above 1 (gold bars) for some others. Change M, df and the correlation of the two-input example on the right.

Directions M (PCR and PLS)

Ridge df(λ)

Two-input example: correlation ρ

Try these

Prostate data: 67 training cases, predictors standardised as in the book. The factor for component j is vjTβ̂ / vjTβ̂ls. PLS follows Algorithm 3.3 and its coefficients are recovered from the sequence of transformations. The two-input example is the population version of Figure 3.18: XTX = [[1, ρ], [ρ, 1]] and true coefficients (4, 2), so least squares is exactly (4, 2).

In the lecture notes