Self-attention ignores the order of its inputs, so a code of the position is added to each embedding. A binary code toggles its right-most bit at every step; the sinusoidal code \(p_{i,2j}=\sin(i/C^{2j/d})\), \(p_{i,2j+1}=\cos(i/C^{2j/d})\) (Equation 15.57) is a real-valued version with geometrically spaced wavelengths. Its key property is that a shift by \(\phi\) is the same rotation \(\mathbf R(\phi)\) of the code at every position, and that the similarity \(\mathbf p_t^{\mathsf T}\mathbf p_{t+\phi}\) depends only on the offset \(\phi\). The second view looks at the other design choice of Section 15.4.3: dot-product scores have variance \(d\), so without the factor \(1/\sqrt d\) the softmax saturates onto a single key as \(d\) grows.
Code view: the heat map shows \(\mathbf P\in\mathbb R^{n\times d}\) (rows are positions); the curve is \(\mathbf p_t^{\mathsf T}\mathbf p_s/(d/2)\) against \(s\) for the reference position \(t\), with the dotted curve \(\frac2d\sum_k\cos(\omega_k(s-t))\), \(\omega_k=C^{-2k/d}\). The rotation error is \(\max|\mathbf p_{t+\phi}-\mathbf R(\phi)\mathbf p_t|\). Score view: 4000 draws of \(\mathbf q\) and \(\mathbf k\) with independent standard normal entries from a seeded generator for the histograms; 1000 draws of one query and \(m\) keys for the mean largest attention weight and entropy.
code/ch15_neural_networks_sequences.py (positional_encoding, pe_rotation, attention, fig_15_25, fig_15_S6).