Take (X, Y) standard bivariate Gaussian with correlation ρ, so that I(X;Y) = −½ ln(1 − ρ²) nats. The upper bound replaces p(y) by q(y) = N(0, s²). The Barber–Agakov bound replaces p(x|y) by q(x|y) = N(ay, τ²). The Donsker–Varadhan, NWJ and InfoNCE bounds use the critic f = λf* + c, where f* is the true log density ratio, so λ = 1 is the perfect critic. InfoNCE also needs a batch of K pairs and can never exceed ln K.
Values are in nats. DV, NWJ, BA and the upper bound are exact integrals (closed forms); InfoNCE is a Monte Carlo average over batches, shown with its standard error. The Donsker–Varadhan bound shown is the standard form with the logarithm of the average over all pairs of independent draws; it does not depend on the offset c, whereas the NWJ bound does.
code/ch05_information_theory.py (upper_bound, ba_bound, dv_bound, nwj_bound, infonce_mc).