Interactive demo · Statistical Learning: Theory and Methods · Chapter 15

Out-of-bag error against test error as a random forest grows

Each tree of a random forest is grown on a bootstrap sample, which leaves out about 37% of the training cases. Predicting every case with only the trees that never saw it gives the out-of-bag (OOB) error, an estimate of the test error that comes for free while the forest is being grown. The data follow the model (15.11): inputs uniform on [0, 1]p, Pr(Y = 1 | x) = 0.9 when x1 + x2 > 1 and 0.1 otherwise, plus any number of pure-noise inputs. The left panel adds trees one at a time; the right panel shows the forest's vote in the plane of the two relevant inputs (noise inputs held at 0.5).

Training sample size N

Noise inputs (2 relevant + this many)

Candidate variables per split m (0 = default ⌊√p⌋)

Number of trees B

Label noise q (Bayes error)

Random seed

Try these

Trees are grown to purity (minimum node size 1) with the Gini index; at each node the variables are taken in a random order and the best split among the first m that vary in the node is used. The forest classifies by majority vote (ties to class 0). The OOB error after b trees is computed over the cases that are out of bag for at least one of them. The test set has 500 points from the same model.

In the lecture notes