Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 14

Committees, bagging and regression trees

The data are noisy values of sin(2πx) (gold curve). A regression tree cuts the x-axis into intervals and predicts the mean of the targets in each; grown deep it follows the noise. Bagging fits the same model to M bootstrap samples (faint lines) and averages them (teal): the committee. The errors are measured against the true function: E_AV is the average error of the members (14.10), E_COM the error of their average (14.11). Averaging always helps, E_COM ≤ E_AV, but by much less than the factor 1/M of (14.14), because members fitted to overlapping bootstrap samples make correlated errors.

Base model

Maximum depth

Pruning λ in C(T) = Σ Q_τ + λ|T|

Polynomial degree

Committee size M

Data

Scoreboard

Try these

E_AV − E_COM = (1/M) ∑_m E_x[(y_m(x) − y_COM(x))²], the spread of the members around their average, so the committee gains exactly as much as its members disagree (Exercise 14.3). Expectations over x are averages over 400 equally spaced points in (0, 1).

In the lecture notes

Chapter 14 notes: Section 14.2 (committees and bagging, E_AV and E_COM, (14.7) to (14.14), Exercises 14.2 and 14.3) and Section 14.4 (tree-based models: greedy growing, the leaf means (14.29), the pruning criterion (14.31) and the impurities (14.32), (14.33)).
Python code: code/ch14_combining_models.py (functions grow_tree, prune_tree, pruning_path, bagging, fit_polynomial, committee_errors); checks in code/checks_ch14.py.