Course · Lecture notes, demonstrations and code

Statistical Learning: Theory and Methods

Mathematical foundations, demonstrations and code

A course that follows Trevor Hastie, Robert Tibshirani and Jerome Friedman's The Elements of Statistical Learning (2nd edition, Springer, 2009) chapter by chapter: linear methods for regression and classification, basis expansions and kernel smoothing, model assessment and inference, additive models and trees, boosting, neural networks and support vector machines, nearest neighbours, unsupervised learning, random forests and ensembles, graphical models and high-dimensional problems.

TrungTin Nguyen · Teaching page · Homepage

What each chapter offers

Lecture notes that fill the gaps. The notes follow the book section by section, with its section, equation, figure and exercise numbers. Wherever the book states a result tersely or leaves it to an exercise, the notes write out the whole argument in a "Filling the gap" box: 162 derivations in all, from the Gauss-Markov theorem and the lasso path to the representer theorem, AIC and BIC, the EM algorithm, AdaBoost as forward stagewise fitting, the SVM dual, PageRank and the graphical lasso. All 208 exercises have worked solutions. Where we found a slip in the book, the notes say so and show the evidence.

The book's figures, reproduced. The Python code rebuilds 296 figures from the book's own data sets (prostate, spam, zip-code digits, vowel, South African heart disease, microarrays and more), one function per figure, with the key algorithms (LARS, backfitting, boosting, the SVM path, the graphical lasso) implemented directly.

Demonstrations on the book's data. 53 browser demonstrations let you move a penalty, a bandwidth, a number of neighbours, a tree size or a number of boosting rounds and see the effect at once. The notes open each one at the case being discussed, and each links back to the notes.

Before you start

Prerequisites. Linear algebra, multivariable calculus, and a first course in probability and statistics (including linear regression).

The book. The book, its data sets and its figures are available from the authors' website. Keep it open next to the notes: section, equation, figure and exercise numbers match.

Companion courses. Many topics overlap the Bayesian and Probabilistic Machine Learning and Foundations of Deep Learning courses. Each notes page lists the overlapping sections and links them.

Chapters

Chapter 1

Introduction

The book's motivating examples (spam, prostate cancer, handwritten digits, DNA microarrays) and the supervised-learning framework: inputs, outputs, loss, risk, training and test error.

4 figures reproduced · 5 gap-filling derivations

Chapter 2

Overview of Supervised Learning

Least squares and nearest neighbours as two extremes, statistical decision theory, the curse of dimensionality, structured regression models and the bias-variance trade-off.

11 figures reproduced · 8 gap-filling derivations · 9 exercises with solutions

Chapter 3

Linear Methods for Regression

Least squares and its sampling theory, the Gauss-Markov theorem, subset selection, ridge regression and the lasso, least angle regression, principal components and partial least squares.

21 figures reproduced · 15 gap-filling derivations · 30 exercises with solutions

Chapter 4

Linear Methods for Classification

Indicator regression and masking, linear and quadratic discriminant analysis, reduced-rank LDA, logistic regression by IRLS, and separating hyperplanes.

15 figures reproduced · 11 gap-filling derivations · 9 exercises with solutions

Chapter 5

Basis Expansions and Regularization

Piecewise polynomials and splines, natural cubic splines, smoothing splines and their degrees of freedom, reproducing kernel Hilbert spaces, wavelets and B-splines.

22 figures reproduced · 12 gap-filling derivations · 19 exercises with solutions

Chapter 6

Kernel Smoothing Methods

Kernel smoothers, local linear and polynomial regression and their boundary bias, local likelihood, kernel density estimation and classification, naive Bayes.

17 figures reproduced · 9 gap-filling derivations · 12 exercises with solutions

Chapter 7

Model Assessment and Selection

Bias, variance and model complexity, the optimism of the training error, Cp, AIC and BIC, minimum description length, the VC dimension, cross-validation and the bootstrap.

14 figures reproduced · 10 gap-filling derivations · 10 exercises with solutions

Chapter 8

Model Inference and Averaging

The bootstrap and maximum likelihood, Bayesian inference, the EM algorithm, MCMC and Gibbs sampling, bagging, model averaging, stacking and bumping.

19 figures reproduced · 9 gap-filling derivations · 7 exercises with solutions

Chapter 9

Additive Models, Trees, and Related Methods

Generalized additive models and backfitting, regression and classification trees, PRIM, MARS, hierarchical mixtures of experts and missing data.

15 figures reproduced · 12 gap-filling derivations · 6 exercises with solutions

Chapter 10

Boosting and Additive Trees

AdaBoost and forward stagewise additive modelling, exponential loss, boosting trees, gradient boosting, regularisation, and interpretation with importance and partial dependence.

21 figures reproduced · 7 gap-filling derivations · 12 exercises with solutions

Chapter 11

Neural Networks

Projection pursuit regression, single-hidden-layer networks, back-propagation, weight decay and starting values, and the ZIP code networks.

16 figures reproduced · 10 gap-filling derivations · 7 exercises with solutions

Chapter 12

Support Vector Machines and Flexible Discriminants

The support vector classifier and its dual, kernels, the hinge loss, the SVM regularisation path, flexible, penalised and mixture discriminant analysis.

15 figures reproduced · 9 gap-filling derivations · 11 exercises with solutions

Chapter 13

Prototype Methods and Nearest-Neighbors

Prototype methods (K-means, LVQ, Gaussian mixtures), nearest-neighbour classifiers and the Cover-Hart bound, tangent distance and adaptive (DANN) metrics.

14 figures reproduced · 5 gap-filling derivations · 8 exercises with solutions

Chapter 14

Unsupervised Learning

Association rules, clustering, self-organising maps, principal components and curves, spectral clustering, kernel and sparse PCA, NMF, ICA, MDS and PageRank.

44 figures reproduced · 15 gap-filling derivations · 25 exercises with solutions

Chapter 15

Random Forests

Random forests: decorrelated trees, out-of-bag error, variable importance, proximity plots, and the analysis of variance and bias.

13 figures reproduced · 6 gap-filling derivations · 7 exercises with solutions

Chapter 16

Ensemble Learning

Ensembles as penalised regression over a dictionary, forward stagewise and the lasso, the bet on sparsity, importance-sampled ensembles and rule ensembles.

9 figures reproduced · 4 gap-filling derivations · 4 exercises with solutions

Chapter 17

Undirected Graphical Models

Markov graphs, Gaussian graphical models, estimation with a known graph, the graphical lasso, and Boltzmann machines.

10 figures reproduced · 5 gap-filling derivations · 12 exercises with solutions

Chapter 18

High-Dimensional Problems: p ≫ N

Problems with p much larger than N: diagonal LDA and nearest shrunken centroids, regularised classifiers, supervised principal components, and the false discovery rate.

16 figures reproduced · 10 gap-filling derivations · 20 exercises with solutions

Shared resources

Python

Code that reproduces the figures

One module per chapter and one function per figure, reading the book's data sets, with readable implementations of the chapter's algorithms.

Companion courses

Bayesian and Probabilistic Machine Learning

The same format for Bishop's Pattern Recognition and Machine Learning. See also Foundations of Deep Learning and Statistical Methods for Modern Biology.

Using the material in class

  1. Before the lecture, students read a section of the book and the matching section of the notes, with the derivations closed.
  2. In the lecture, state a result, ask the class to predict what it implies, then open the demonstration at the linked preset and test the prediction.
  3. In the tutorial, open the derivations one at a time and work through the exercises at the end of each chapter.
  4. For assignments, students change one setting in the Python code (a penalty, a bandwidth, a tree size, a data set) and explain what changes in the figure.