The inputs are the 67 training cases of the prostate cancer data (8 standardised inputs, optionally followed by pure-noise inputs). The "truth" is the least-squares fit of lpsa on the 8 real inputs, and new responses y = f + σε are simulated again and again with the inputs held fixed. For each model size we average the training error, the in-sample error Errin (new responses at the same inputs), Cp = err + 2dσ²/N and BIC, and we estimate the optimism and the degrees of freedom ∑ Cov(ŷi, yi)/σ² directly. With a fixed order of entry the optimism is exactly 2dσ²/N; when forward stepwise searches for the inputs, the effective number of parameters exceeds d and Cp becomes too optimistic.
Size k = number of inputs; every model has an intercept, so d = k + 1. Fixed order: lcavol, lweight, svi, lbph, pgg45, lcp, age, gleason, then the noise inputs. Forward stepwise adds, at each step, the input that most reduces the residual sum of squares of the current simulated y. σ is a multiple of the residual standard deviation of the full fit, 0.712. The oracle size minimises Errin, which needs the truth.
code/ch07_model_assessment.py (optimism_design, forward_path, optimism_demo).