Write any linear fit in the coordinates of the principal components of the inputs. Least squares uses each component with factor 1; ridge regression multiplies component j by dj2/(dj2 + λ); principal components regression keeps the first M components and discards the rest; partial least squares, which also looks at the response, applies factors that are mostly below 1 for the low-variance components but can be above 1 (gold bars) for some others. Change M, df and the correlation of the two-input example on the right.
Prostate data: 67 training cases, predictors standardised as in the book. The factor for component j is vjTβ̂ / vjTβ̂ls. PLS follows Algorithm 3.3 and its coefficients are recovered from the sequence of transformations. The two-input example is the population version of Figure 3.18: XTX = [[1, ρ], [ρ, 1]] and true coefficients (4, 2), so least squares is exactly (4, 2).
code/ch03_linear_regression.py (pcr, pls, two_input_paths, Figures 3.17 and 3.18); the coefficients shown here equal the Python ones.