Interactive demo · Probabilistic Machine Learning · Chapter 18

AdaBoost with decision stumps

Boosting fits a sequence of weak classifiers, each to a re-weighted copy of the data: examples the ensemble gets wrong receive a larger weight before the next round. Slide the number of rounds M. The area of each point is its weight for the next round, the shaded plane is the sign of the weighted vote of the first M stumps. The curves show how the training error is squeezed under the exponential loss, which equals the product of the round factors 2√(errm(1−errm)), while the test error flattens or rises.

Rounds M

Weak learner

Flipped training labels (%)

Try these

Algorithm 18.1: weights start at 1/N; round m fits a tree Fm to the weighted data, computes its weighted error errm, sets αm = log((1 − errm)/errm), multiplies the weights of the points it misclassifies by eαm and renormalises. The classifier is sgn(Σm αmFm(x)). Data: 200 training and 1000 test points of the noisy two-moons problem.

In the lecture notes