Interactive demo · Foundations of Deep Learning · Chapter 11

Markov models of text: what the order k buys, and what it costs

An autoregressive model writes p(x1, …, xN) = ∏n p(xn | x1, …, xn−1) (11.42). A Markov chain of order k keeps only the last k symbols, so its conditional is one table with Kk(K − 1) free entries. Here K = 28 (space, full stop, a to z) and the table is fitted by counting in the first 90% of Alice's Adventures in Wonderland; the last 10% is held out. Small k forgets too much; large k has far more parameters than characters to fit them.

Order k

Smoothing p = (n + α) / (N + Kα)

Unseen contexts

Generate

Loading the text and counting…

Try these

In the lecture notes