The data are a mixture of 16 equally weighted Gaussians (standard deviation 0.05) on a 4 × 4 grid. Seven small networks, all with the same initialisation and 32 hidden units, were trained for 14 000 generator iterations with Algorithm 26.2; each variant changes one ingredient of the game. Drag the iteration slider or press play to replay the generated samples (400 of the 2000 used for the statistics). A mode counts as covered when at least half of its fair share of the samples (1/32) lies within four standard deviations of it, and a sample is of high quality when it lies within four standard deviations of some mode. The baseline discovers a few modes, loses some of them again and hops between the others; the variants differ in how many modes they recover and how sharply.
Snapshots are stored at iterations 0, 500, 1000, 2000, 3000, 4000, 6000, 8000, 10 000, 12 000 and 14 000; the curves are evaluated every 250 iterations. Generator and discriminator: two hidden layers of 32 ReLU units, a two-dimensional latent code, Adam with learning rate 10−3 and momentum 0.5 unless stated, minibatch 64 unless stated. “Non-saturating” uses −log D(G(z)) for the generator, “saturating” log(1 − D(G(z))). R1 adds (γ/2)E∥∇xD∥2 on real data with γ = 10. The Wasserstein GAN uses a gradient penalty of weight 1 and five critic steps per generator step. The MMD generator has no discriminator: it minimises a sum of four Gaussian-kernel squared MMDs on batches of 256. One run per variant (seed 0): the ranking of variants is an illustration, not a benchmark.
code/ch26_gans.py (train_gan, GAN_CONFIGS, mode_stats); data data/ch26-modes.json.