Thirty points with five standard Gaussian features, pairwise correlation 0.95, and Pr(Y = 1 | x1 > 0.5) = 0.8, Pr(Y = 1 | x1 ≤ 0.5) = 0.2, so the Bayes error is 0.2. Because the features are nearly interchangeable, a tree grown on a bootstrap sample often splits on a different feature at a different point: the tree is a high-variance procedure. Bagging grows a tree on each of B bootstrap samples and lets them vote (or averages their leaf proportions). Under squared error, averaging can only help; under 0–1 loss it helps at inputs where a single tree is right more than half of the time and hurts where it is right less than half of the time. The lower-right panel shows both, over many training sets.
Trees are grown on the deviance with the stopping rules of the R package tree (a node is split only if it has at least minsize cases and its deviance is at least 1% of the root's; each child keeps at least minsize/2 cases); no pruning. The test set has 2000 points. The lower-right panel uses 25 fresh training sets with B = 50 each and the first 500 test inputs; "agrees with the Bayes rule" means predicting class 1 exactly when x1 > 0.5, and the expected error at an input is 0.2 + 0.6 × Pr(disagree).
code/ch08_model_inference_averaging.py (Tree, sim_corr_data, bagging_curves, bagging_example).