The response is the log of prostate-specific antigen (lpsa) in 97 men, split by the book into 67 training men (filled dots) and 30 test men (open rings). Fit a straight line by least squares, or average the k training responses whose input is closest (the k-nearest-neighbour rule), and compare the average squared error on the data used for fitting with the error on the 30 men that were not used. The training error always improves as the fit becomes more flexible; the test error does not.
Errors are mean squared errors, (1/n)∑(yi − ŷi)2, over the 67 training or the 30 test men; every fit uses the training men only. When several training men are tied at the k-th smallest distance, the nearest-neighbour average includes all of them. With “all eight inputs” the left panel plots the response against the fitted value, and the right panel compares the three models.
code/ch01_introduction.py (knn_regress, ols_fit, prostate_fits, supplementary Figure 1.S1).