A representation X of n examples has p neurons with a few strong directions. The second representation Y is a noisy copy of X after a transformation of your choice, or independent data. Six similarity indexes of Section 32.2.2 are computed in the page: linear centred kernel alignment (CKA), the mean squared and the mean canonical correlation, SVCCA, projection-weighted CCA and the variance-weighted R2LR. Rotations and permutations should change nothing, a rescaling of single neurons separates CKA from CCA, and when there are more neurons than examples CCA reports perfect similarity for unrelated data.
The data are a fixed table of 80 examples with 60 neurons (a few latent factors plus noise). Y = T(X + noise × sd(X) × E) where T is the chosen map and E a fixed table of standard normal noise; the first n examples and p neurons are used. Everything is centred before use.
code/ch32_representation_learning.py (linear_cka, cca, svcca, pwcca, r2_lr_cca); data: data/ch32-reps.json.