Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 5

Variational bounds on mutual information

Take (X, Y) standard bivariate Gaussian with correlation ρ, so that I(X;Y) = −½ ln(1 − ρ²) nats. The upper bound replaces p(y) by q(y) = N(0, s²). The Barber–Agakov bound replaces p(x|y) by q(x|y) = N(ay, τ²). The Donsker–Varadhan, NWJ and InfoNCE bounds use the critic f = λf* + c, where f* is the true log density ratio, so λ = 1 is the perfect critic. InfoNCE also needs a batch of K pairs and can never exceed ln K.

The pair

Critic for DV, NWJ, InfoNCE

Variational distributions

Try these

Values are in nats. DV, NWJ, BA and the upper bound are exact integrals (closed forms); InfoNCE is a Monte Carlo average over batches, shown with its standard error. The Donsker–Varadhan bound shown is the standard form with the logarithm of the average over all pairs of independent draws; it does not depend on the offset c, whereas the NWJ bound does.

In the lecture notes