Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 16

Attention is a soft lookup

Twelve or more keys sit on the unit circle, each carrying a value yi = sin 2θi + 0.3 cos 5θi. A query q is another point on the circle. Attention scores every key by β q·ki, turns the scores into weights with a softmax and returns the weighted average of the values. The weights are drawn as the size of each key. Because all keys have the same length, this is exactly Gaussian kernel regression: β is the inverse of the kernel width.

Number of keys n

Sharpness β (inverse temperature)

Query angle φ (degrees)

Mask

Try these

Scores a = β q·k, weights αi = softmax(a)i as in (16.10), output ∑i αiyi as in (16.8). With β = 0 every key gets weight 1/n; as β grows the output becomes the value of the nearest key. The causal mask lets a query see only keys whose angle is not larger than its own, as in a decoder-only transformer.

In the lecture notes