Interactive demo · Bayesian and Probabilistic Machine Learning · Advanced topics

Autoregressive models: generating an image one pixel at a time

An autoregressive model writes the joint distribution of an 8 by 8 binary image as a product of conditionals in raster order: p(x) = ∏i p(xi | x<i). Each pixel is a logistic function of its bias and of a few already-generated neighbours. Sampling fills the grid pixel by pixel; the log-likelihood of the image is the sum of the logs of the conditionals (the chain rule). The causal model is a valid density. The leaky model lets a pixel read the pixel below it, which is not yet generated: its conditionals no longer define a joint distribution, and the toy check shows the total mass is not one.

Model

Weights seed

Sampling temperature

Generation

Try these

Weights: each pixel has a bias near −0.7, a strong weight (about 1.2) on its left and upper neighbours, and small weights on the other causal neighbours of a window of 2 rows and 2 columns above and to the left. All come from a seeded generator, so the images look like smooth blobs. The leaky model adds weight 2.5 on the pixel directly below. The leaky model has no ancestral sampler, so it is sampled in three raster sweeps: each sweep reads the pixel below from the previous sweep. The toy check uses a 2 by 2 image with the same rules and the same seed.

In the lecture notes