Interactive demo · Statistical Learning: Theory and Methods · Chapter 7

Cross-validation the wrong way and the right way

Each data set has N = 50 samples in two classes of 25 and p Gaussian inputs. The recipe: (1) keep the m inputs most correlated with the class labels, (2) classify by the nearest neighbour on those inputs, (3) estimate the error by K-fold cross-validation. Done the wrong way, step (1) uses all 50 samples and only steps (2)–(3) are cross-validated; done the right way, the screening is repeated inside every fold, without the held-out samples. With no signal at all the true error is 50%: the wrong way reports a few percent.

Inputs p

Inputs kept by the screening m

Folds K

Signal: mean shift of class 2 in 10 inputs

Screening rule

Simulate

Try these

Labels alternate between the classes, so each class has 25 samples. Folds are random (one shuffle per data set). The test error uses 200 new samples drawn from the same distribution and the classifier built from all 50 samples. Correlation screening keeps the largest |correlation|; variance screening keeps the largest variances and never looks at the labels.

In the lecture notes