Each method produces one model of every size. Best subset fits all 2p models to find the smallest RSS of each size; forward and backward stepwise selection fit only 1 + p(p+1)/2 and are greedy, so they sometimes miss it. Training RSS always prefers the biggest model, so the size is chosen by a criterion that charges for each variable: Cp charges 2σ̂2, BIC charges log(n)σ̂2. Slide the charge and watch the chosen size fall.
Criterion (RSS + c·d·σ̂2)/n with σ̂2 = RSSfull/(n − p − 1): c = 2 is Cp (6.2) and AIC, c = log n is BIC (6.3). Adjusted R2 is (6.4). Cross-validation is 10-fold, with best subset selection redone inside every fold (computed once in Python; the folds are fixed). The coefficients in the table are refitted by least squares in your browser.
code/ch06_model_selection_regularization.py (the searches and the data of this page) and code/ch06_lab.py.