A language model outputs a probability for every token of its vocabulary; generation must turn it into one choice. Here the distribution is real: the relative frequencies of the words that follow a given context in Alice's Adventures in Wonderland (Lewis Carroll, 1865). Temperature rescales the logits, yi ∝ exp(ai/T) (12.35); top-k keeps the k most probable words; nucleus (top-p) sampling keeps the smallest set of most probable words whose total probability reaches p. The bars compare the original distribution (outline) with the one actually sampled (filled). Below, greedy and beam search look for a whole high-probability sequence.
T = 0 is greedy selection. The rules are applied in the order temperature, top-k, nucleus, and the result is renormalised. Draws use the inverse cumulative distribution with a seeded generator, so a seed always gives the same words.
A bigram model over the 60 commonest words of the novel, p(next | previous) from counts plus 0.5. The most probable continuation of length L maximises the product (12.34). Greedy search takes the best word at each step; beam search keeps the B best partial sequences; for a bigram model the exact optimum is found by dynamic programming (Viterbi), which is impossible for a transformer, whose next-word distribution depends on the whole prefix.
code/ch12_transformers.py (sampling_dist, draw, beam_search, viterbi_bigram, export_sampling, supplementary Figure 12.S6); the distributions, draws and searches here equal the Python ones.