Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 32

Comparing two representations: CKA and CCA

A representation X of n examples has p neurons with a few strong directions. The second representation Y is a noisy copy of X after a transformation of your choice, or independent data. Six similarity indexes of Section 32.2.2 are computed in the page: linear centred kernel alignment (CKA), the mean squared and the mean canonical correlation, SVCCA, projection-weighted CCA and the variance-weighted R2LR. Rotations and permutations should change nothing, a rescaling of single neurons separates CKA from CCA, and when there are more neurons than examples CCA reports perfect similarity for unrelated data.

Representations

Transformation of the copy

Try these

The data are a fixed table of 80 examples with 60 neurons (a few latent factors plus noise). Y = T(X + noise × sd(X) × E) where T is the chosen map and E a fixed table of standard normal noise; the first n examples and p neurons are used. Everything is centred before use.

In the lecture notes