Interactive demo · Foundations of Deep Learning · Chapter 12

Sampling strategies for the next token

A language model outputs a probability for every token of its vocabulary; generation must turn it into one choice. Here the distribution is real: the relative frequencies of the words that follow a given context in Alice's Adventures in Wonderland (Lewis Carroll, 1865). Temperature rescales the logits, yi ∝ exp(ai/T) (12.35); top-k keeps the k most probable words; nucleus (top-p) sampling keeps the smallest set of most probable words whose total probability reaches p. The bars compare the original distribution (outline) with the one actually sampled (filled). Below, greedy and beam search look for a whole high-probability sequence.

Context (the words before)

Temperature T

Top-k (0 = off)

Nucleus p (1 = off)

Random seed for the draws

Try these

T = 0 is greedy selection. The rules are applied in the order temperature, top-k, nucleus, and the result is renormalised. Draws use the inverse cumulative distribution with a seeded generator, so a seed always gives the same words.

In the lecture notes