Interactive demo · Probabilistic Machine Learning · Chapter 15

Decoding a character-level GRU

A gated recurrent unit with 80 hidden units, trained for 1500 steps on Alice's Adventures in Wonderland (28 symbols), defines a distribution \(p(y_t\mid y_{1:t-1})\) over the next character. Greedy decoding takes the most probable character at every step and soon falls into a loop. Sampling draws from \(p^{1/T}\): a low temperature \(T\) is nearly greedy, a high one approaches uniform noise (the entropy increases with \(T\)). Beam search keeps the \(K\) most probable prefixes and returns a more probable string than greedy decoding. The colour of each generated character shows the probability the model gave it. Switch to the toy tree view to watch beam search prune candidates on the four-symbol example of Figure 15.12 and the five-symbol model of Figure 15.13.

View

Prefix

Method

Characters generated n

Temperature T

Beam width K

Inspect step t

Try these

The network runs here in double precision with the weights rounded to four decimals. Sampling uses a seeded random generator, so "Draw again" gives a new but reproducible text; greedy and beam search are deterministic. In the toy view the candidates at each step are the \(KV\) extensions of the kept prefixes, ranked by probability; a prefix that has produced eos is finished and carried over unchanged.

In the lecture notes