Course · Lecture notes, demonstrations and code

Probability and Statistical Inference

Mathematical foundations, demonstrations and code

A course that follows Larry Wasserman's All of Statistics: A Concise Course in Statistical Inference (Springer, 2004) chapter by chapter: probability, random variables, inequalities and convergence; estimation, the bootstrap, likelihood, testing, Bayesian inference and decision theory; and regression, multivariate and categorical models, causal inference, graphical models, nonparametric smoothing, classification, stochastic processes and simulation.

TrungTin Nguyen · Teaching page · Homepage

What each chapter offers

Lecture notes that prove what the book states. The book is concise by design and quotes many results without proof. The notes follow it section by section, with its numbering, and supply the argument in a "Filling the gap" box: 369 derivations in all, from the axioms and Hoeffding's inequality to the central limit theorem, Glivenko-Cantelli, asymptotic normality of the MLE, the Neyman-Pearson lemma, Bernstein-von Mises, Stein's paradox, Hammersley-Clifford, Markov chain convergence and the Metropolis-Hastings algorithm. All 257 exercises have worked solutions. Where we found a slip in the book, the notes say so, mark the ones already in the author's errata, and show the evidence.

The book's figures, reproduced. The Python code rebuilds 237 figures from the book's own data sets (nerve impulses, Old Faithful, coronary risk, glass, the cosmic microwave background and more) and from simulation, one function per figure.

Demonstrations you can run. 73 browser demonstrations let you change a sample size, a prior, a bandwidth, a significance level or a step size and see the effect at once. The notes open each one at the case being discussed, and each links back to the notes.

Before you start

Prerequisites. Calculus, linear algebra and some exposure to probability; no measure theory is assumed (the measure-theoretic background is supplied in the notes where it matters).

The book. The book's data sets and the author's errata are on its website. Keep the book open next to the notes: section, equation, figure and exercise numbers match.

Companion courses. This book overlaps every other course on this site. Each notes page lists the overlapping sections of Bayesian and Probabilistic Machine Learning, Foundations of Deep Learning, Statistical Methods for Modern Biology, Statistical Learning: Theory and Methods and Applied Statistical Learning in R and Python, and links them.

Chapters

Chapter 1

Probability

Sample spaces and events, the axioms of probability, independence, conditional probability and Bayes' theorem, with the measure-theoretic background.

8 figures reproduced · 13 gap-filling derivations · 23 exercises with solutions

Chapter 2

Random Variables

Random variables, distribution functions and quantiles, the standard discrete and continuous families, joint, marginal and conditional distributions, and transformations.

12 figures reproduced · 5 gap-filling derivations · 21 exercises with solutions

Chapter 3

Expectation

Expectation, variance and covariance, the sample mean and variance, the multivariate normal, conditional expectation and moment generating functions.

6 figures reproduced · 13 gap-filling derivations · 24 exercises with solutions

Chapter 4

Inequalities

Markov, Chebyshev, Hoeffding, Mill's, Cauchy-Schwarz and Jensen inequalities, with full proofs and the confidence intervals they give.

7 figures reproduced · 10 gap-filling derivations · 7 exercises with solutions

Chapter 5

Convergence of Random Variables

Modes of convergence and the implications between them, the laws of large numbers, the central limit theorem, Berry-Esseen and the delta method.

7 figures reproduced · 16 gap-filling derivations · 16 exercises with solutions

Chapter 6

Models, Statistical Inference and Learning

Parametric and nonparametric models, point estimation, bias, variance and mean squared error, consistency, confidence sets and the language of testing.

6 figures reproduced · 15 gap-filling derivations · 3 exercises with solutions

Chapter 7

Estimating the cdf and Statistical Functionals

The empirical distribution function, Glivenko-Cantelli and the DKW inequality, plug-in estimators of statistical functionals and the influence function.

7 figures reproduced · 13 gap-filling derivations · 10 exercises with solutions

Chapter 8

The Bootstrap

Bootstrap variance estimation, normal, pivotal and percentile intervals, the jackknife, and when the bootstrap works.

8 figures reproduced · 12 gap-filling derivations · 8 exercises with solutions

Chapter 9

Parametric Inference

The method of moments, maximum likelihood, consistency, asymptotic normality and efficiency of the MLE, the delta method, sufficiency, exponential families, Newton-Raphson and EM.

10 figures reproduced · 24 gap-filling derivations · 10 exercises with solutions

Chapter 10

Hypothesis Testing and p-values

The Wald, likelihood ratio, chi-squared and permutation tests, p-values, the Neyman-Pearson lemma, and multiple testing.

11 figures reproduced · 17 gap-filling derivations · 16 exercises with solutions

Chapter 11

Bayesian Inference

Priors and posteriors, conjugate models, posterior simulation, Bernstein-von Mises, Jeffreys' prior, Bayes factors, and the strengths and weaknesses of the Bayesian approach.

7 figures reproduced · 20 gap-filling derivations · 8 exercises with solutions

Chapter 12

Statistical Decision Theory

Loss and risk, Bayes rules, minimax rules, admissibility, and Stein's paradox with the James-Stein estimator.

9 figures reproduced · 19 gap-filling derivations · 6 exercises with solutions

Chapter 13

Linear and Logistic Regression

Simple and multiple linear regression, least squares and its sampling theory, prediction intervals, model selection (Cp, AIC, BIC, cross-validation) and logistic regression.

9 figures reproduced · 15 gap-filling derivations · 11 exercises with solutions

Chapter 14

Multivariate Models

Random vectors, the multivariate normal and its estimation, the sample correlation and Fisher's z, Hotelling's T-squared, and the multinomial.

6 figures reproduced · 19 gap-filling derivations · 6 exercises with solutions

Chapter 15

Inference About Independence

Testing and measuring dependence: the odds ratio, chi-squared tests for tables, correlation tests and the two-sample Kolmogorov-Smirnov test.

5 figures reproduced · 12 gap-filling derivations · 7 exercises with solutions

Chapter 16

Causal Inference

Potential outcomes, association versus causation, randomisation, confounding and adjustment, and Simpson's paradox.

7 figures reproduced · 10 gap-filling derivations · 5 exercises with solutions

Chapter 17

Directed Graphs and Conditional Independence

Conditional independence, directed acyclic graphs and their Markov properties, d-separation, Markov equivalence, and interventions.

18 figures reproduced · 13 gap-filling derivations · 9 exercises with solutions

Chapter 18

Undirected Graphs

Undirected graphs, the pairwise, local and global Markov properties, cliques and potentials, Hammersley-Clifford, and Gaussian graphical models.

18 figures reproduced · 11 gap-filling derivations · 5 exercises with solutions

Chapter 19

Log-Linear Models

Log-linear models for contingency tables, graphical and hierarchical models, iterative proportional fitting, and model selection.

9 figures reproduced · 12 gap-filling derivations · 6 exercises with solutions

Chapter 20

Nonparametric Curve Estimation

Histograms, kernel density estimation and nonparametric regression: bias, variance, optimal bandwidths and rates, cross-validation and confidence bands.

14 figures reproduced · 20 gap-filling derivations · 10 exercises with solutions

Chapter 21

Smoothing Using Orthogonal Functions

Orthogonal series: the cosine basis and Legendre polynomials, series density and regression estimators, confidence bands, and Haar wavelets with thresholding.

20 figures reproduced · 17 gap-filling derivations · 12 exercises with solutions

Chapter 22

Classification

The Bayes rule, linear and quadratic discriminant analysis, logistic regression, naive Bayes, trees, error-rate estimation, VC theory, support vector machines and boosting.

14 figures reproduced · 20 gap-filling derivations · 13 exercises with solutions

Chapter 23

Probability Redux: Stochastic Processes

Markov chains (classification of states, stationary distributions, convergence, reversibility, inference) and Poisson processes.

8 figures reproduced · 24 gap-filling derivations · 16 exercises with solutions

Chapter 24

Simulation Methods

Monte Carlo integration, importance sampling, the Metropolis-Hastings algorithm, the Gibbs sampler and convergence diagnostics.

11 figures reproduced · 19 gap-filling derivations · 5 exercises with solutions

Shared resources

Python

Code that reproduces the figures

One module per chapter and one function per figure, with readable implementations of the bootstrap, Newton-Raphson and EM, kernel and series smoothers, iterative proportional fitting and MCMC.

Companion courses

Statistical Learning: Theory and Methods

Regression, classification and smoothing in more depth. See also Bayesian and Probabilistic Machine Learning and the first-year notes on estimation and testing.

Using the material in class

  1. Before the lecture, students read a section of the book and the matching section of the notes, with the derivations closed.
  2. In the lecture, state a result, ask the class to predict what it implies, then open the demonstration at the linked preset and test the prediction.
  3. In the tutorial, open the derivations one at a time and work through the exercises at the end of each chapter.
  4. For assignments, students change one setting in the Python code (a sample size, a prior, a bandwidth, a proposal) and explain what changes in the figure.