Interactive demo · Probability and Statistical Inference · Chapter 13

Scoring and searching: Cp, AIC, BIC and cross-validation on every subset of covariates

With k covariates there are 2k linear models (the intercept is always in). Each score of Section 13.6 adds a complexity penalty to the training error: Mallows' Cp adds 2|S|σ̂2, BIC adds |S|σ̂2log n, and leave-one-out cross-validation needs no penalty at all because each point is predicted by a fit that never saw it. Panel (a) shows the chosen score for every model, the best model of each size and what a greedy search finds; panel (b) puts Cp, BIC and cross-validation on one scale.

Data

Score

Search

Try these

|S| counts the columns of the model, intercept included. σ̂2 is RSS/(n − k − 1) of the full model. Book's AIC (13.27) and BIC (13.31): log-likelihood with σ fixed at σ̂, minus |S| or |S|(log n)/2, larger is better. R's AIC and BIC: n log(RSS/n) + 2|S| or + |S| log n, σ estimated in each model, smaller is better. LOO CV by the shortcut (13.29). Zheng-Loh: rank the covariates by |Wj| in the full model and minimise RSS(j) + jσ̂2log n.

In the lecture notes