Both methods regress the response on M linear combinations of the standardised predictors. PCR takes the M directions in which the predictors vary most, ignoring the response; PLS builds its directions from the covariance with the response. So with few components PLS explains more of Salary and less of the predictors, and with all 19 both are least squares. Bottom right: change what drives sales in the simulated cities and watch the first PLS direction turn while the principal component does not move.
Hitters: the 19 predictors are standardised (1/n standard deviation); PCR uses the eigenvectors of ZTZ, PLS the NIPALS algorithm for one response (Algorithm 6.4). The cross-validation errors are 10-fold, with fixed folds, the standardisation and the components recomputed in every fold (computed once in Python). Cities: sales = 2 + a(pop − 38) + b(ad − 16) + noise, with the noise fixed; both directions are computed from the centred, unstandardised data so they can be drawn in the original units.
code/ch06_model_selection_regularization.py (Figures 6.14–6.21 and the data of this page) and code/ch06_lab.py.