A decoder-only transformer (Section 12.3.1): 39 characters, embeddings of dimension D = 64 plus sinusoidal positions (12.25), two pre-norm layers (12.21), (12.23) of four-head masked attention and a ReLU MLP, a context of 64 characters, 105 000 parameters. It was trained in two minutes on one CPU core by next-character cross-entropy on Alice's Adventures in Wonderland (Lewis Carroll, 1865) and is exported with its exact weights; the page computes every forward pass itself. Each generated character is highlighted in proportion to the probability the model gave it: the fainter the highlight, the more surprising the character.
Loading the weights...
Each step feeds the last 64 characters through the network and draws the next character from the rule chosen: temperature (12.35), then top-k, then nucleus, renormalised; greedy takes the most probable character; beam search keeps the B most probable continuations. Held-out cross-entropy of this model: 1.42 nats per character (perplexity 4.1, against 39 for uniform guessing and 4.8 for the best n-gram model).
code/ch12_transformers.py (train_gpt, gpt_forward, generate, beam_search, export_char_model); the logits, samples and beams here equal the Python ones. Weights: data/ch12-char-gpt.json.