Twelve or more keys sit on the unit circle, each carrying a value yi = sin 2θi + 0.3 cos 5θi. A query q is another point on the circle. Attention scores every key by β q·ki, turns the scores into weights with a softmax and returns the weighted average of the values. The weights are drawn as the size of each key. Because all keys have the same length, this is exactly Gaussian kernel regression: β is the inverse of the kernel width.
Scores a = β q·k, weights αi = softmax(a)i as in (16.10), output ∑i αiyi as in (16.8). With β = 0 every key gets weight 1/n; as β grows the output becomes the value of the nearest key. The causal mask lets a query see only keys whose angle is not larger than its own, as in a decoder-only transformer.
code/ch16_deep_neural_networks.py (attention, circle_attention_curve); see the code index.