An average of B trees, each with variance σ2 and pairwise correlation ρ, has variance ρσ2 + (1 − ρ)σ2/B. More trees remove only the second term; the first is set by the correlation. A random forest lowers ρ by letting each split choose among only m randomly drawn inputs, at the price of a larger bias. The numbers below come from the regression simulation (15.8) of the book, Y = (X1 + … + X50)/√50 + ε, with 100 training cases, repeated over 500 training sets (or a sparse variant in which only 5 inputs matter), and evaluated at 600 test points.
Regression trees with minimum node size 5, grown on bootstrap samples. For every training set, 10 trees were grown for each m; the forest's variance and squared bias for B = ∞ are estimated without bias from the spread between and within training sets (see the notes, Filling the gap 15.5). In the lower-right panel the dots are the simulated variances of the average of the first b trees (b ≤ 10), the curve is the formula. Averages are over the 600 test points.
code/ch15_random_forests.py (decorrelation_sim, decor_summary, ridge_on_sim).