Two classes C1 and C2 generate a single input x from Gaussians. The best rule uses the posterior probabilities p(Ck|x): assign x to the class with the smallest expected loss, and refuse to decide when even the best choice is too risky. The shaded areas under the joint densities are the probability of a mistake. The other two views show the entropy of a discrete distribution (largest for the uniform distribution) and the Kullback-Leibler divergence between two Gaussians (never negative, not symmetric).
Class-conditional densities are N(μk, σk2) and the joint density is p(x, Ck) = p(Ck) p(x|Ck). Areas are computed on a grid of 2001 points over [−5, 8] by the trapezoidal rule, the same grid and rule as code/ch01_introduction.py. The loss of a correct decision is 0; the reject option costs λ = 1 − θ, so with 0–1 loss it rejects exactly where the larger posterior is at most θ (Section 1.5.3). In the entropy view the bins are indexed 0, …, 29 and the probabilities are proportional to a Gaussian bump (or two, or a flat floor with a spike).
code/ch01_introduction.py (functions bayes_decision, expected_loss, reject_decision, error_probability, entropy, kl_discrete, kl_gauss, discretised_gaussian_hist; Figures 1.24, 1.26, 1.27, 1.30, 1.31) and the checks in code/checks_ch01.py.