Mathematical foundations, demonstrations and code
A course that follows Kevin Murphy's Probabilistic Machine Learning: An Introduction (MIT Press, 2022) chapter by chapter: the probability, statistics, decision theory, information theory, linear algebra and optimisation that machine learning rests on; linear models; deep networks for tabular data, images and sequences; nearest neighbours, kernels and trees; and learning with fewer labels, dimensionality reduction, clustering, recommender systems and graph embeddings.
Lecture notes that prove what the book states. The book covers the whole of modern machine learning and leaves many steps to the reader. The notes follow it section by section, with its numbering, and supply the argument in a "Filling the gap" box: 437 derivations in all, from Gaussian conditioning and the exponential family to the spectral theorem, convergence rates of gradient methods, backpropagation, the representer theorem, boosting as gradient descent, the Cover-Hart bound and the PCA and t-SNE derivations. Every exercise has a worked solution, and chapters without exercises in the book get practice problems of our own (107 in all). Where we found a slip in the book, the notes say so and show the evidence.
The book's figures, reproduced. The Python code rebuilds 416 figures, one function per figure, from bundled data sets and from simulations; where the book uses data that cannot be obtained offline, a stand-in is used and the caption says so. Deep networks are trained at CPU scale.
Demonstrations you can run. 79 browser demonstrations let you change a prior, a learning rate, a kernel width, a regularisation strength, a number of neighbours or a sample size and see the effect at once. The notes open each one at the case being discussed, and each links back to the notes.
Prerequisites. Calculus, linear algebra and a first course in probability. Chapters 2 to 8 review the probability, statistics, linear algebra and optimisation the rest of the book needs; the companion course Probability and Statistical Inference covers the statistics in more depth.
The book. The book is available from its web page. Keep it open next to the notes: section, equation, figure and exercise numbers match.
Companion courses. Each notes page lists the overlapping sections of Bayesian and Probabilistic Machine Learning, Foundations of Deep Learning, Statistical Learning: Theory and Methods, Applied Statistical Learning in R and Python, Probability and Statistical Inference and the other courses on this site, and links them.
What machine learning is: supervised, unsupervised and reinforcement learning, overfitting and generalisation, the no-free-lunch theorem, and preprocessing of discrete, text and missing data.
11 figures reproduced · 12 gap-filling derivations · 4 exercises with solutions
Probability as a calculus of uncertainty: Bayes' rule, the Bernoulli, binomial, categorical, Gaussian, Student, Laplace, beta and gamma families, transformations of random variables and the central limit theorem.
25 figures reproduced · 8 gap-filling derivations · 12 exercises with solutions
Joint distributions, covariance and correlation, Simpson's paradox, the multivariate Gaussian with its conditionals and Bayes rule, mixture models, the exponential family and directed graphical models.
13 figures reproduced · 18 gap-filling derivations · 8 exercises with solutions
Maximum likelihood, empirical risk minimisation, regularisation and cross-validation, Bayesian inference with conjugate priors, credible intervals, the Laplace approximation, and the frequentist view through the bootstrap and bias-variance trade-off.
27 figures reproduced · 23 gap-filling derivations · 9 exercises with solutions
Bayesian decision theory, ROC and precision-recall curves, Bayesian model comparison with Bayes factors, BIC, AIC and WAIC, frequentist risk and admissibility, empirical risk minimisation and hypothesis testing.
15 figures reproduced · 26 gap-filling derivations · 4 exercises with solutions
Entropy, cross entropy, KL divergence and mutual information, their inequalities and chain rules, the data processing inequality, Fano's inequality, and the maximal information coefficient.
5 figures reproduced · 11 gap-filling derivations · 4 exercises with solutions
Vector spaces, norms, matrix properties, eigenvalue and singular value decompositions, the power method, whitening, LU, QR and Cholesky factorisations, least squares, and matrix calculus.
9 figures reproduced · 36 gap-filling derivations · 2 exercises with solutions
First- and second-order optimisation: convexity, gradient descent and line search, momentum, Newton and quasi-Newton methods, stochastic gradient methods with variance reduction and Adam, constrained and proximal optimisation, and EM.
31 figures reproduced · 18 gap-filling derivations · 2 exercises with solutions
Gaussian discriminant analysis, linear and quadratic decision boundaries, Fisher's linear discriminant, naive Bayes classifiers, and the generative versus discriminative comparison.
13 figures reproduced · 14 gap-filling derivations · 1 exercises with solutions
Binary and multinomial logistic regression, their convex losses, gradient, Newton and IRLS fitting, regularisation, robust variants, and Bayesian logistic regression with the Laplace approximation.
16 figures reproduced · 16 gap-filling derivations · 3 exercises with solutions
Least squares and its geometry, ridge, lasso, elastic net and group lasso, the lasso path and coordinate descent, splines and additive models, robust regression, and Bayesian linear regression.
25 figures reproduced · 29 gap-filling derivations · 6 exercises with solutions
Generalized linear models: the exponential dispersion family, canonical and non-canonical links, maximum likelihood by IRLS, deviance, and Poisson regression for claim counts.
4 figures reproduced · 6 gap-filling derivations · 5 exercises with solutions
Multilayer perceptrons for tabular data: universal approximation, backpropagation and automatic differentiation, vanishing gradients, initialisation, regularisation, double descent, RBF networks and mixtures of experts.
19 figures reproduced · 19 gap-filling derivations · 1 exercises with solutions
Convolutional networks: convolution as a structured matrix, pooling, normalisation layers, classic architectures, residual and dense networks, segmentation, detection, and generating images by inverting a trained network.
28 figures reproduced · 21 gap-filling derivations · 6 exercises with solutions
Recurrent networks, backpropagation through time, gated units, decoding by greedy and beam search, one-dimensional convolutions, attention, transformers, positional encodings, efficient attention and pretrained language models.
17 figures reproduced · 22 gap-filling derivations · 8 exercises with solutions
Nearest-neighbour classification and its error, the curse of dimensionality, metric learning (LMNN, NCA, deep metric learning), and kernel density estimation and kernel regression.
16 figures reproduced · 16 gap-filling derivations · 6 exercises with solutions
Mercer kernels, Gaussian processes for regression and classification, marginal likelihood, scalable approximations and random features, support vector machines for classification and regression, and sparse vector machines.
21 figures reproduced · 23 gap-filling derivations · 1 exercises with solutions
Classification and regression trees, ensembles, bagging and random forests, boosting from AdaBoost to gradient boosting and XGBoost, and interpreting tree ensembles.
18 figures reproduced · 19 gap-filling derivations · 8 exercises with solutions
Data augmentation, transfer learning and adapters, contrastive pretraining, semi-supervised learning, active learning with information gain and BALD, meta-learning, few-shot learning and weak supervision.
15 figures reproduced · 21 gap-filling derivations · 1 exercises with solutions
PCA and its probabilistic and factor-analysis versions, autoencoders and variational autoencoders, manifold learning (MDS, Isomap, kernel PCA, LLE, Laplacian eigenmaps, t-SNE), and word embeddings.
38 figures reproduced · 28 gap-filling derivations · 9 exercises with solutions
Measuring cluster quality, hierarchical agglomerative clustering, K-means and K-means++, choosing the number of clusters, mixture-model clustering, spectral clustering and biclustering.
27 figures reproduced · 20 gap-filling derivations · 4 exercises with solutions
Explicit and implicit feedback, matrix factorisation by ALS and SGD, probabilistic matrix factorisation, Bayesian personalised ranking, factorisation machines, cold start, and bandit exploration.
13 figures reproduced · 14 gap-filling derivations
Shallow graph embeddings from distances, matrix factorisation and random walks, label propagation, graph neural networks and graph convolutions, hyperbolic embeddings and unsupervised graph learning.
10 figures reproduced · 17 gap-filling derivations · 3 exercises with solutions
One module per chapter and one function per figure, with readable implementations of EM, gradient and Newton methods, logistic and linear regression, backpropagation, Gaussian processes, support vector machines, trees and boosting, PCA and t-SNE, K-means and matrix factorisation.
The same subject through Bishop's Pattern Recognition and Machine Learning. See also Foundations of Deep Learning for Chapters 13 to 15 and Statistical Learning: Theory and Methods for Chapters 11, 16 to 18 and 21.