Forward stagewise regression (Algorithm 16.1) builds a model by many tiny moves: find the input most correlated with the current residual and change its coefficient by ε in the direction of that correlation. Nothing is ever refitted, yet the coefficient paths look like the lasso's. On the left is the exact lasso path; on the right the staircase of ε-steps and, dashed, its limit FS0 as ε → 0, computed exactly. When every lasso coefficient moves monotonically (the prostate data) the three coincide; with blocks of strongly correlated inputs (a small version of the book's simulation) the lasso paths swing back and forth while stagewise ones stay (almost) monotone. The bottom panel shows the test error along both paths.
Inputs are centred and scaled to unit variance; the lasso and FS0 paths are computed exactly by least angle regression (with the lasso modification, or with the sign-constrained direction of FS0). Simulation: 60 observations, 200 inputs in 10 blocks of 20, one nonzero N(0,1) coefficient per block, noise variance 0.72 times the signal variance; its test error is the exact E(f̂ − f)² over the population. Prostate: mean squared error on the 30 test cases. Colours: one per input (prostate) or one per block (simulation).
code/ch16_ensemble_learning.py (lars_gram, nnls_gram, fs_eps_gram, demo_block_data); the paths drawn here are those of the Python functions on the same data.