Interactive demo · Probabilistic Machine Learning · Chapter 6

Correlation versus binned mutual information

Each point cloud has a clear relationship between x and y. The squared correlation R2 only sees the straight-line part; the mutual information, estimated by cutting each axis into B equal-width bins and using the plug-in formula on the resulting table, sees any dependence. The estimate depends on B and on the sample size: with many bins and few points it is large even for independent data (the dashed curve is the same estimate after pairing each x with the y of another point, which destroys the relationship).

Relationship

Noise level

Sample size

Bins per axis B

Try these

Mutual information in bits, from the equal-width B × B table over the range of the data. NMI is I divided by the smaller marginal entropy of the binned data. The relationships are generated from stored random numbers, so every visitor sees the same points.

In the lecture notes