Interactive demo · Probabilistic Machine Learning · Chapter 6

Joint and conditional entropy, and mutual information

Enter non-negative weights for a joint table p(x, y) with four values of X (columns) and four of Y (rows); they are normalised to sum to one. The bar picture is the information diagram: the joint entropy H(X,Y) splits into H(X | Y) + I(X;Y) + H(Y | X), and the marginal entropies are the first two pieces and the last two pieces. The mutual information is the KL divergence from the product of the marginals to the joint.

Weights (rows y, columns x)

Try these

Logarithms are base 2 (bits). A table is rescaled by its total, so only ratios matter. Rows or columns of zeros are allowed.

In the lecture notes