Interactive demo · Probabilistic Machine Learning · Chapter 18

Bagging and random forests on two moons

A fully grown tree fits the training points exactly and its boundary is jagged: a high-variance estimator. Bagging fits many trees, each to a bootstrap resample, and averages their class probabilities; a random forest also lets every split look at only a few randomly chosen features, which makes the trees less alike. The left plot shows the averaged decision surface for the chosen number of trees, the right plot the test error and the out-of-bag (OOB) error as trees are added.

Method

Number of trees B

Training points n (of 250)

Features tried per split m (forest only)

Random seed

Try these

Training data: 250 points of the noisy two-moons problem (noise 0.3); test data: 750 further points from the same distribution. The ensemble predicts the class with the larger average leaf probability. The OOB error of a training point uses only the trees whose bootstrap sample missed it (about 37% of them).

In the lecture notes