Interactive demo · Applied Statistical Learning in R and Python · Chapter 6

Best subset against stepwise selection, with Cp, BIC and adjusted R2

Each method produces one model of every size. Best subset fits all 2p models to find the smallest RSS of each size; forward and backward stepwise selection fit only 1 + p(p+1)/2 and are greedy, so they sometimes miss it. Training RSS always prefers the biggest model, so the size is chosen by a criterion that charges for each variable: Cp charges 2σ̂2, BIC charges log(n)σ̂2. Slide the charge and watch the chosen size fall.

Data

Method

Criterion for the size

Charge per variable c (in units of σ̂2)

Size shown, k

Try these

Criterion (RSS + c·d·σ̂2)/n with σ̂2 = RSSfull/(n − p − 1): c = 2 is Cp (6.2) and AIC, c = log n is BIC (6.3). Adjusted R2 is (6.4). Cross-validation is 10-fold, with best subset selection redone inside every fold (computed once in Python; the folds are fixed). The coefficients in the table are refitted by least squares in your browser.

In the lecture notes