Interactive demo · Probabilistic Machine Learning · Chapter 3

Correlation, hidden groups and Simpson's paradox

The Pearson correlation \(\rho=\operatorname{Cov}[X,Y]/\sqrt{\mathbb V[X]\mathbb V[Y]}\) measures the degree of linearity of a cloud, and the regression slope is \(\operatorname{Cov}[X,Y]/\mathbb V[X]\). Tick any combination of the three iris species: within each species sepal width rises with sepal length, but the setosa flowers have short, wide sepals, and once they are pooled with the others the overall trend points downwards. A trend seen in every group can vanish or reverse in the mixture.

Species shown

Lines

Try these

Data: the 150 flowers of Fisher's iris data set (50 per species), first two measurements, in centimetres. The correlation and slope are computed from the flowers currently shown, with the divisor n (the ratio is unaffected). The mixture is simply the union of the ticked groups, so the number of flowers per group is fixed at 50.

In the lecture notes