Interactive demo · Probabilistic Machine Learning · Chapter 4

Bootstrap, posterior and confidence intervals for a proportion

You observe k heads in N tosses. The frequentist asks how the estimate k/N would vary over repeated data sets: the bootstrap approximates that sampling distribution by resampling the data (top left). The Bayesian reports the posterior of θ instead (top right); for a weak prior the two look alike, so the bootstrap is a “poor man's posterior”. The lower panel is the exact probability that each 95% interval contains the true θ, as θ varies: the Wald interval p ± 1.96√(p(1−p)/N) can fall far below 95%, and is degenerate when k = 0 or N. A confidence level describes the procedure over repeated data, not the probability that θ lies in the one interval you computed.

Tosses N

Heads observed k

True probability θ* (for the coverage panel)

Bootstrap / posterior draws B

Try these

The bootstrap resamples the N observed tosses with replacement B times and records the proportion of heads each time. The posterior uses the uniform Beta(1,1) prior, so it is Beta(k+1, N−k+1); its draws come from the inverse cdf. Coverage is exact (a sum over the binomial distribution of k), not simulated. Wald uses the interval (p, p) when k = 0 or N. Jeffreys is the central credible interval of Beta(k+½, N−k+½).

In the lecture notes