Mathematical foundations, demonstrations and code
A course that follows Trevor Hastie, Robert Tibshirani and Jerome Friedman's The Elements of Statistical Learning (2nd edition, Springer, 2009) chapter by chapter: linear methods for regression and classification, basis expansions and kernel smoothing, model assessment and inference, additive models and trees, boosting, neural networks and support vector machines, nearest neighbours, unsupervised learning, random forests and ensembles, graphical models and high-dimensional problems.
Lecture notes that fill the gaps. The notes follow the book section by section, with its section, equation, figure and exercise numbers. Wherever the book states a result tersely or leaves it to an exercise, the notes write out the whole argument in a "Filling the gap" box: 162 derivations in all, from the Gauss-Markov theorem and the lasso path to the representer theorem, AIC and BIC, the EM algorithm, AdaBoost as forward stagewise fitting, the SVM dual, PageRank and the graphical lasso. All 208 exercises have worked solutions. Where we found a slip in the book, the notes say so and show the evidence.
The book's figures, reproduced. The Python code rebuilds 296 figures from the book's own data sets (prostate, spam, zip-code digits, vowel, South African heart disease, microarrays and more), one function per figure, with the key algorithms (LARS, backfitting, boosting, the SVM path, the graphical lasso) implemented directly.
Demonstrations on the book's data. 53 browser demonstrations let you move a penalty, a bandwidth, a number of neighbours, a tree size or a number of boosting rounds and see the effect at once. The notes open each one at the case being discussed, and each links back to the notes.
Prerequisites. Linear algebra, multivariable calculus, and a first course in probability and statistics (including linear regression).
The book. The book, its data sets and its figures are available from the authors' website. Keep it open next to the notes: section, equation, figure and exercise numbers match.
Companion courses. Many topics overlap the Bayesian and Probabilistic Machine Learning and Foundations of Deep Learning courses. Each notes page lists the overlapping sections and links them.
The book's motivating examples (spam, prostate cancer, handwritten digits, DNA microarrays) and the supervised-learning framework: inputs, outputs, loss, risk, training and test error.
4 figures reproduced · 5 gap-filling derivations
Least squares and nearest neighbours as two extremes, statistical decision theory, the curse of dimensionality, structured regression models and the bias-variance trade-off.
11 figures reproduced · 8 gap-filling derivations · 9 exercises with solutions
Least squares and its sampling theory, the Gauss-Markov theorem, subset selection, ridge regression and the lasso, least angle regression, principal components and partial least squares.
21 figures reproduced · 15 gap-filling derivations · 30 exercises with solutions
Indicator regression and masking, linear and quadratic discriminant analysis, reduced-rank LDA, logistic regression by IRLS, and separating hyperplanes.
15 figures reproduced · 11 gap-filling derivations · 9 exercises with solutions
Piecewise polynomials and splines, natural cubic splines, smoothing splines and their degrees of freedom, reproducing kernel Hilbert spaces, wavelets and B-splines.
22 figures reproduced · 12 gap-filling derivations · 19 exercises with solutions
Kernel smoothers, local linear and polynomial regression and their boundary bias, local likelihood, kernel density estimation and classification, naive Bayes.
17 figures reproduced · 9 gap-filling derivations · 12 exercises with solutions
Bias, variance and model complexity, the optimism of the training error, Cp, AIC and BIC, minimum description length, the VC dimension, cross-validation and the bootstrap.
14 figures reproduced · 10 gap-filling derivations · 10 exercises with solutions
The bootstrap and maximum likelihood, Bayesian inference, the EM algorithm, MCMC and Gibbs sampling, bagging, model averaging, stacking and bumping.
19 figures reproduced · 9 gap-filling derivations · 7 exercises with solutions
Generalized additive models and backfitting, regression and classification trees, PRIM, MARS, hierarchical mixtures of experts and missing data.
15 figures reproduced · 12 gap-filling derivations · 6 exercises with solutions
AdaBoost and forward stagewise additive modelling, exponential loss, boosting trees, gradient boosting, regularisation, and interpretation with importance and partial dependence.
21 figures reproduced · 7 gap-filling derivations · 12 exercises with solutions
Projection pursuit regression, single-hidden-layer networks, back-propagation, weight decay and starting values, and the ZIP code networks.
16 figures reproduced · 10 gap-filling derivations · 7 exercises with solutions
The support vector classifier and its dual, kernels, the hinge loss, the SVM regularisation path, flexible, penalised and mixture discriminant analysis.
15 figures reproduced · 9 gap-filling derivations · 11 exercises with solutions
Prototype methods (K-means, LVQ, Gaussian mixtures), nearest-neighbour classifiers and the Cover-Hart bound, tangent distance and adaptive (DANN) metrics.
14 figures reproduced · 5 gap-filling derivations · 8 exercises with solutions
Association rules, clustering, self-organising maps, principal components and curves, spectral clustering, kernel and sparse PCA, NMF, ICA, MDS and PageRank.
44 figures reproduced · 15 gap-filling derivations · 25 exercises with solutions
Random forests: decorrelated trees, out-of-bag error, variable importance, proximity plots, and the analysis of variance and bias.
13 figures reproduced · 6 gap-filling derivations · 7 exercises with solutions
Ensembles as penalised regression over a dictionary, forward stagewise and the lasso, the bet on sparsity, importance-sampled ensembles and rule ensembles.
9 figures reproduced · 4 gap-filling derivations · 4 exercises with solutions
Markov graphs, Gaussian graphical models, estimation with a known graph, the graphical lasso, and Boltzmann machines.
10 figures reproduced · 5 gap-filling derivations · 12 exercises with solutions
Problems with p much larger than N: diagonal LDA and nearest shrunken centroids, regularised classifiers, supervised principal components, and the false discovery rate.
16 figures reproduced · 10 gap-filling derivations · 20 exercises with solutions
One module per chapter and one function per figure, reading the book's data sets, with readable implementations of the chapter's algorithms.
The same format for Bishop's Pattern Recognition and Machine Learning. See also Foundations of Deep Learning and Statistical Methods for Modern Biology.