Interactive demo · Foundations of Deep Learning · Chapter 12

A tiny character transformer, running in your browser

A decoder-only transformer (Section 12.3.1): 39 characters, embeddings of dimension D = 64 plus sinusoidal positions (12.25), two pre-norm layers (12.21), (12.23) of four-head masked attention and a ReLU MLP, a context of 64 characters, 105 000 parameters. It was trained in two minutes on one CPU core by next-character cross-entropy on Alice's Adventures in Wonderland (Lewis Carroll, 1865) and is exported with its exact weights; the page computes every forward pass itself. Each generated character is highlighted in proportion to the probability the model gave it: the fainter the highlight, the more surprising the character.

Loading the weights...

Prompt

Decoding

Temperature T

Top-k (0 = off)

Nucleus p (1 = off)

Seed

Beam width B

Characters to generate

Attention head shown

Try these

Each step feeds the last 64 characters through the network and draws the next character from the rule chosen: temperature (12.35), then top-k, then nucleus, renormalised; greedy takes the most probable character; beam search keeps the B most probable continuations. Held-out cross-entropy of this model: 1.42 nats per character (perplexity 4.1, against 39 for uniform guessing and 4.8 for the best n-gram model).

In the lecture notes