Interactive demo · Statistical Methods for Modern Biology · Chapter 5

How many clusters? Gap statistic, Calinski-Harabasz index and silhouettes

Every clustering algorithm returns k groups when asked for k. Here PAM is run for k = 1, ..., Kmax and four criteria judge the results. The gap statistic compares log Wk with its average over B reference data sets drawn uniformly over a box that contains the data (no clusters), and picks k with a one-standard-error rule. The Calinski-Harabasz index is the F ratio of between- to within-cluster variation, and the silhouette width of a point compares its average distance to its own cluster with that to the nearest other cluster.

Data

Largest k tried

Reference data sets B

Reference box

Power of the distances in Wk

Rule for choosing k

Show the clustering with k =

Scoreboard

Try these

Wk = Σl (1/2nl) Σi<j in cluster l dijp, exactly as in cluster::clusGap: its default power p = 1 sums plain Euclidean distances, while p = 2 gives half the within-cluster sum of squares used by Tibshirani et al. (2001). The reference box is aligned either with the principal axes of the data (spaceH0 = "scaledPCA", the default) or with the original coordinates. The 50-dimensional embryo data are drawn on their first two principal components.

In the lecture notes