With k covariates there are 2k linear models (the intercept is always in). Each score of Section 13.6 adds a complexity penalty to the training error: Mallows' Cp adds 2|S|σ̂2, BIC adds |S|σ̂2log n, and leave-one-out cross-validation needs no penalty at all because each point is predicted by a fit that never saw it. Panel (a) shows the chosen score for every model, the best model of each size and what a greedy search finds; panel (b) puts Cp, BIC and cross-validation on one scale.
|S| counts the columns of the model, intercept included. σ̂2 is RSS/(n − k − 1) of the full model. Book's AIC (13.27) and BIC (13.31): log-likelihood with σ fixed at σ̂, minus |S| or |S|(log n)/2, larger is better. R's AIC and BIC: n log(RSS/n) + 2|S| or + |S| log n, σ estimated in each model, smaller is better. LOO CV by the shortcut (13.29). Zheng-Loh: rank the covariates by |Wj| in the full model and minimise RSS(j) + jσ̂2log n.
code/ch13_regression.py (scores_for, subset_scores, stepwise, zheng_loh; Figure 13.S3).