Interactive demo · Applied Statistical Learning in R and Python · Chapter 8

Boosting regression trees on the Hitters data

Log salary of the first 200 players (with a salary) is learned slowly: each new tree, with d splits, is fitted to the current residuals, shrunk by λ and added to the fit (Algorithm 8.2, (8.10)–(8.12)); the other 63 players are the test set. Small λ needs many trees, large λ overfits quickly, and stumps (d = 1) build an additive model, one function of one input at a time.

Shrinkage λ

Splits per tree d (interaction depth)

Number of trees B

Start from

Try these

Each tree is grown best-first: the split with the largest drop in residual sum of squares among all current leaves is made, d times, with at least 10 players per leaf (as R's gbm(), but without its random half-samples, so the result is exactly reproducible). "Zero" is Algorithm 8.2 as printed; "the mean" is what gbm() and scikit-learn do.

In the lecture notes