Interactive demo · Statistical Learning: Theory and Methods · Chapter 10

Gradient boosting: shrinkage, subsampling and robust loss

The data are noisy values of a wave with a jump (gold curve). Gradient boosting starts from the best constant and at every iteration computes the pseudo-residuals ri = −∂L(yi, f(xi))/∂f (the residuals for squared error, their signs for absolute error, clipped residuals for Huber loss), fits a regression tree with J leaves to them by least squares, replaces each leaf value by the best constant for the loss in that leaf, and adds the tree times the shrinkage factor ν. With subsampling each tree sees only a random fraction η of the points. The lower panel follows the error of the fit against the true curve: watch how ν and η move its minimum, and how the loss decides what outliers do.

Loss

Leaves per tree J

Shrinkage ν

Subsample fraction η

Outliers

Training points N

Iteration m

Data seed

Try these

Leaf values: the mean of the residuals (squared error), their median (absolute error), or for Huber loss the median plus the mean of the clipped deviations from it, with δ the 0.9 quantile of the absolute residuals at that iteration. The error against the truth is the average of |f̂(x) − f(x)| over 500 equally spaced points. Outliers add ±(3 to 6) to a response.

In the lecture notes