Interactive demo · Statistical Learning: Theory and Methods · Chapter 10

AdaBoost on the nested spheres

Ten inputs are independent standard normals and the class is Y = +1 when the point lies outside the sphere ∑xj2 = 9.34 (the median of a χ210), −1 inside. A stump splits one input at one threshold; on its own it is barely better than a coin. AdaBoost.M1 fits stumps one after another to reweighted data, multiplying the weight of every misclassified point by eαm with αm = log((1 − errm)/errm), and classifies by the sign of f(x) = ∑ βmGm(x), βm = αm/2. Every stump depends on one input, so f is a sum of one function per input: the lower left panel shows one of them against the quadratic xj2 that the true boundary is built from. The lower right panel shows where the weights have gone.

Base learner and leaf values

Data

Training points N

Training labels flipped at random

Horizon (rounds computed)

Round m

Coordinate shown j

Data seed

Try these

The test set has 5000 points drawn from the same distribution, with correct labels. "Real-valued" stumps choose the split that minimises the weighted exponential loss and give each side the value ½ log(W+/W−) of its weighted class totals, (10.31)–(10.32) with the factor ½. In the overlapping setting class −1 is N(0, I) and class +1 is N(0, I) conditioned on ∑xj2 > 12; the Bayes error is then 0.1425.

In the lecture notes