Mathematical foundations, demonstrations and code
A course that follows An Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani (with Jonathan Taylor for the Python edition) chapter by chapter, in both editions at once: the R edition (2nd edition, Springer, 2021) and the Python edition (Springer, 2023). Each chapter has one set of notes, and its lab is written twice, in R and in Python, shown side by side.
Lecture notes that fill the gaps. The notes follow the book section by section, with its section, figure and exercise numbers, and write out the mathematics the book leaves aside in "Filling the gap" boxes: 150 derivations in all. For every topic that the graduate book treats in more depth, a "Deeper" link points to the matching section of Statistical Learning: Theory and Methods.
Every lab in R and in Python. The labs are written for this course in base R with the standard packages, and in Python with pandas, statsmodels, scikit-learn and PyTorch. The notes show the two versions side by side with their output, and say exactly where the two languages give different numbers and why. All 142 exercises have worked solutions, the applied ones in both languages.
Figures and demonstrations. The code reproduces 176 of the book's figures from its data, and 38 browser demonstrations let you move a tuning parameter and see the effect at once. The notes open each one at the case being discussed.
Prerequisites. A first course in statistics and some algebra; no prior programming is assumed beyond what the Chapter 1 and 2 labs teach.
The book. Both editions, their data and their labs are available from the authors' website. Keep the book open next to the notes: section, figure and exercise numbers match.
Companion courses. Statistical Learning: Theory and Methods covers the same methods at graduate level; the Bayesian and Probabilistic Machine Learning, Foundations of Deep Learning and Statistical Methods for Modern Biology courses are linked where they overlap.
The book's three motivating data sets (Wage, Smarket, NCI60), supervised and unsupervised learning, notation, and a first tour of R and Python.
4 figures reproduced · 6 gap-filling derivations
What statistical learning estimates, prediction versus inference, parametric and non-parametric methods, training and test error, the bias-variance trade-off, the Bayes classifier and K-nearest neighbours.
17 figures reproduced · 9 gap-filling derivations · 10 exercises with solutions
Simple and multiple linear regression, standard errors, tests and intervals, qualitative predictors and interactions, diagnostics (leverage, outliers, collinearity), and regression against KNN.
20 figures reproduced · 15 gap-filling derivations · 15 exercises with solutions
Logistic regression, linear and quadratic discriminant analysis, naive Bayes, KNN, the confusion matrix and the ROC curve, and Poisson regression as a generalized linear model.
15 figures reproduced · 11 gap-filling derivations · 16 exercises with solutions
The validation set, leave-one-out and k-fold cross-validation (with the least-squares shortcut), and the bootstrap.
9 figures reproduced · 9 gap-filling derivations · 11 exercises with solutions
Best subset and stepwise selection, Cp, AIC, BIC and adjusted R-squared, ridge regression and the lasso, principal components and partial least squares, and problems with p larger than n.
24 figures reproduced · 15 gap-filling derivations · 11 exercises with solutions
Polynomials and step functions, regression and natural splines, smoothing splines, local regression and generalized additive models.
14 figures reproduced · 12 gap-filling derivations · 12 exercises with solutions
Regression and classification trees and cost-complexity pruning, bagging, random forests, boosting and Bayesian additive regression trees.
9 figures reproduced · 12 gap-filling derivations · 12 exercises with solutions
The maximal margin classifier, the support vector classifier, kernels and the support vector machine, multiclass SVMs, and the link with logistic regression.
12 figures reproduced · 8 gap-filling derivations · 8 exercises with solutions
Single and multilayer networks, convolutional and recurrent networks, backpropagation, stochastic gradient descent, dropout, and interpolation and double descent.
11 figures reproduced · 12 gap-filling derivations · 13 exercises with solutions
Censoring, the Kaplan-Meier estimator, the log-rank test, proportional hazards and the Cox model, shrinkage for the Cox model, and measures of model performance.
11 figures reproduced · 17 gap-filling derivations · 11 exercises with solutions
Principal components and matrix completion, K-means and hierarchical clustering, and practical choices in clustering.
19 figures reproduced · 10 gap-filling derivations · 13 exercises with solutions
Hypothesis tests and p-values, the family-wise error rate (Bonferroni, Holm, Tukey, Scheffe), the false discovery rate and Benjamini-Hochberg, and permutation approaches.
11 figures reproduced · 14 gap-filling derivations · 10 exercises with solutions
For each chapter, the lab in R and in Python, and a Python module that reproduces the chapter's figures.
The same methods with the full theory, following The Elements of Statistical Learning.