Interactive demo · Statistical Learning: Theory and Methods · Chapter 7

The .632 and .632+ bootstrap against cross-validation

A k-nearest-neighbour classifier is trained on N points of the two-class mixture of Chapter 2 (or on points whose labels carry no information). Its true error on new data is known exactly (the class probabilities of the mixture are known), so every estimate can be compared with the truth. The naive bootstrap tests on points the bootstrap sample has seen; the leave-one-out bootstrap avoids that but trains on only about 63.2% distinct points; the .632 estimator pulls it toward the training error, which overshoots for an overfitting rule such as 1-NN; the .632+ estimator measures the overfitting and pulls less.

Neighbours k

Training-set size N

Labels

Bootstrap samples B

Simulate

Try these

Points: 20 Gaussian clusters with variance 0.2 around the means of the Chapter 2 mixture, 10 per class; labels alternate, so the classes have equal size. "No information": the cluster is drawn from all 20 whatever the label, so the label is independent of x and every rule has error 1/2. True error: the average over 5000 new points of the probability that the rule misclassifies them. 10-fold CV uses ten contiguous blocks. Odd k only, so votes never tie.

In the lecture notes