Interactive demo · Probabilistic Machine Learning · Chapter 5

Bayes factors, BIC and p-values for a coin

Two models for N coin tosses: M0, a fair coin (θ = 1/2), and M1, any coin with θ ~ Beta(a, a). The Bayes factor integrates θ out and therefore pays an Occam penalty for the flexibility of M1; the BIC approximates it; the p-value ignores the alternative altogether. Fix how many standard deviations from 1/2 the head count lies (z) and watch what happens as N grows.

Tosses N

Head count, in standard deviations z = (N1 − N/2) / (√N / 2)

Prior of the free coin Beta(a, a)

Prior probability of the fair coin p(M0)

Significant trials (second readout)

Try these

The head count is rounded to an integer. Bayes factor B0,1 = p(D|M0)/p(D|M1) = 2−N / B(a+N1, a+N0) × B(a, a), which does not depend on the order of the tosses; the p-value is the exact two-sided binomial test (the sum of all outcomes no more probable than the observed one). The two last sliders are the fraction of ineffective drugs (prior of H0) and the type II error β, with α = 0.05.

In the lecture notes