Interactive demo · Probability and Statistical Inference · Chapter 11

The posterior becomes Normal: the Bernstein–von Mises theorem

Theorem 11.5 says that, for large n, the posterior is close to the Normal distribution centred at the maximum likelihood estimate θ̂ with standard deviation sê = 1/√(n I(θ̂)), whatever the (smooth, positive) prior. Top left: the exact posterior (teal) and that Normal (gold, dashed) after n observations, with the Wald interval θ̂ ± z sê shaded under the posterior. Top right: their total variation distance as n grows, on log scales. Bottom: the posterior probability of the Wald interval, which the theorem says tends to 1 − α. Switch to the log scale (log-odds or log-rate) to see the Bayesian delta method at work.

Data and model

Prior

Scale of the parameter

Sample size n

True success probability p

Level 1 − α

Try these

Coin tosses: toss k is a success when the k-th uniform of the seeded generator is below p; the first n tosses are used. Nerve data: the first n of the 799 waiting times of the book's nerve.dat, modelled as exponential with rate λ. The total variation distance is 1 − ∫ min(posterior, Normal), computed with the trapezoidal rule on 2001 points spanning θ̂ ± 12 sê. In the log scale the Normal is N(log-odds or log-rate of the MLE, sê2 g′(θ̂)2). Curves use about 60 sample sizes evenly spaced on a log scale; they skip n with no success or no failure, where sê = 0.

In the lecture notes