Each data set has N = 50 samples in two classes of 25 and p Gaussian inputs. The recipe: (1) keep the m inputs most correlated with the class labels, (2) classify by the nearest neighbour on those inputs, (3) estimate the error by K-fold cross-validation. Done the wrong way, step (1) uses all 50 samples and only steps (2)–(3) are cross-validated; done the right way, the screening is repeated inside every fold, without the held-out samples. With no signal at all the true error is 50%: the wrong way reports a few percent.
Labels alternate between the classes, so each class has 25 samples. Folds are random (one shuffle per data set). The test error uses 200 new samples drawn from the same distribution and the classifier built from all 50 samples. Correlation screening keeps the largest |correlation|; variance screening keeps the largest variances and never looks at the labels.
code/ch07_model_assessment.py (screening_demo_run, screening_study, screen_cv).