An ensemble is built in two stages. First a dictionary of trees is grown by Algorithm 16.2: each tree is fitted to the current residuals on a random fraction η of the training data, and the running model moves by ν times the tree (ν = 0 and η < 1 is bagging, η = 1 is gradient boosting, in between is stochastic gradient boosting). Then the trees' own weights are thrown away and refitted by a lasso path (16.9), which keeps only some of the trees. The data come from the book's simulation (16.13): 500 training and 500 test points with 100 uniform inputs, of which 35 matter. Compare the generator's test error as trees are added (gold) with the post-processed error against the number of trees the lasso keeps (teal).
Trees: least squares, grown best-first to J terminal nodes with at least 5 points each. Lasso: squared error, intercept, the trees' outputs unstandardised, 50 penalties from the largest useful one down by a factor 1000, coordinate descent with warm starts. Test error: the mean of (f̂ − f)² over the 500 test inputs against the true function (noise variance 1.69 is not included). The penalty is chosen here with the test data, as in the book's Figures 16.6–16.8; in practice it would be chosen by cross-validation.
code/ch16_ensemble_learning.py (demo_isle_data, DemoTree, isle_demo, post_process, lasso_cd_gram); the trees and errors here are those of the Python functions on the same data.