Interactive demo · Probabilistic Machine Learning · Chapter 18

Gradient boosting of regression trees

Gradient boosting is gradient descent in function space: each round fits a small regression tree to the negative gradient of the loss at the current fit (the residuals, for squared error) and adds a shrunken copy of it. Choose the loss, the tree depth and the learning rate ν, then slide the number of rounds M. The top plot is the ensemble, the middle plot the residuals and the M-th tree that is fitted to them, the bottom plot the training and test error. The “XGBoost” option adds the second-order leaf weights and the penalties λ and γ of Section 18.5.5.2.

Loss

Rounds M

Learning rate ν

Tree depth

Leaf penalty λ (XGBoost)

Split penalty γ (XGBoost)

Training data

Try these

Data: y = 3x² plus noise (sd 0.05) on 100 points, 200 test points. The outlier option moves five training responses by about ±0.5. Squared loss: residual y − f, leaf value the mean residual. Absolute loss: pseudo-residual sgn(y − f), leaf value the median residual. XGBoost: g = f − y, h = 1, leaf weight −G/(H + λ), split gain (18.56).

In the lecture notes