Interactive demo · Probabilistic Machine Learning · Chapter 15

Positional encoding and attention scores

Self-attention ignores the order of its inputs, so a code of the position is added to each embedding. A binary code toggles its right-most bit at every step; the sinusoidal code \(p_{i,2j}=\sin(i/C^{2j/d})\), \(p_{i,2j+1}=\cos(i/C^{2j/d})\) (Equation 15.57) is a real-valued version with geometrically spaced wavelengths. Its key property is that a shift by \(\phi\) is the same rotation \(\mathbf R(\phi)\) of the code at every position, and that the similarity \(\mathbf p_t^{\mathsf T}\mathbf p_{t+\phi}\) depends only on the offset \(\phi\). The second view looks at the other design choice of Section 15.4.3: dot-product scores have variance \(d\), so without the factor \(1/\sqrt d\) the softmax saturates onto a single key as \(d\) grows.

View

Code

Positions n

Dimensions d (bits for the binary code)

Constant C = 10c

Reference position t and offset φ

Try these

Code view: the heat map shows \(\mathbf P\in\mathbb R^{n\times d}\) (rows are positions); the curve is \(\mathbf p_t^{\mathsf T}\mathbf p_s/(d/2)\) against \(s\) for the reference position \(t\), with the dotted curve \(\frac2d\sum_k\cos(\omega_k(s-t))\), \(\omega_k=C^{-2k/d}\). The rotation error is \(\max|\mathbf p_{t+\phi}-\mathbf R(\phi)\mathbf p_t|\). Score view: 4000 draws of \(\mathbf q\) and \(\mathbf k\) with independent standard normal entries from a seeded generator for the histograms; 1000 draws of one query and \(m\) keys for the mean largest attention weight and entropy.

In the lecture notes