A generator network turns random noise into samples; a discriminator network tries to tell those samples from real data; each is trained against the other. Because the generator never needs a likelihood, it can be any differentiable map, and the discriminator supplies the training signal that a likelihood would otherwise provide. Section 17.1 sets up the game, shows that with unlimited capacity its solution is the data distribution, and explains why training it is hard: the gradients can vanish, the game can cycle, and the generator can collapse onto a few modes. The remedies (a non-saturating loss, least squares, Wasserstein distances) each come from a different way of measuring how far apart two distributions are. Section 17.2 looks at image GANs, their latent spaces and CycleGAN, which learns to translate between two collections of images without paired examples.
Ch. 1 Deep learning revolution Ch. 2 Probabilities Ch. 3 Standard distributions Ch. 4 Regression Ch. 5 Classification Ch. 6 Deep neural networks Ch. 7 Gradient descent Ch. 8 Backpropagation Ch. 9 Regularization Ch. 10 Convolutional networks Ch. 11 Structured distributions Ch. 12 Transformers Ch. 13 Graph neural networks Ch. 14 Sampling Ch. 15 Discrete latent variables Ch. 16 Continuous latent variables Ch. 17 GANs Ch. 18 Normalizing flows Ch. 19 Autoencoders Ch. 20 Diffusion models Appendices A to C
They follow the book. Sections carry the book's numbers (17.1, 17.1.2, ...) and every displayed equation that appears in the book carries the book's own equation number, e.g. (17.15). Equations without a number are steps that the book skips.
Filling the gaps. The chapter is short and states most of its theory in words or leaves it to the exercises. Wherever it says a result "can be shown", gives an argument informally, or leaves it out, an orange Filling the gap box or a proof gives the full argument. Proofs start hidden: open one with its bar, or use the button below. Every numerical claim has been confirmed by computation.
Figures and demonstrations. Of the ten figures, five are computed here: Figures 17.2 and 17.3 directly, and Figures 17.6, 17.9 and 17.10 (which show large published models) at small scale, with two-dimensional data and MNIST digits. The other five are diagrams of architectures and information flow and are not reproduced. Eight supplementary figures (17.S1 to 17.S8, our own) illustrate the gaps that are filled. The ▶ Demo boxes open an interactive demonstration already set to the case being discussed.
C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer (2024), Chapter 17, pages 533–545. Section, equation, figure and exercise numbers below refer to this book. Further reading is listed in the references; the book's website is www.bishopbook.com. There is no official solution for the exercises of this chapter; all solutions below are my own.
The explanations, derivations, figures and code are my own; equations are cited by their numbers in the book.
GANs are not in Bishop (2006), so this chapter is treated in full here. The tools it uses are covered in the companion course Bayesian and Probabilistic Machine Learning, and its advanced module covers the other deep generative models with which GANs are usually compared.
Notation. \(\pd\) is the data density and \(\pG\) the density of \(\bx=\bg(\bz,\bw)\) when \(\bz\sim p(\bz)\); when we write \(p\) and \(q\) for short, \(p=\pd\) and \(q=\pG\). \(d(\bx,\bphi)\in(0,1)\) is the discriminator, \(E\) the error it minimises and the generator maximises (17.6), \(d^\star\) the optimal discriminator, and \(M=(p+q)/2\). \(\JS\) is the Jensen–Shannon divergence (Definition 17.2), \(\W\) the Wasserstein-1 distance (Definition 17.3), \(\sigma\) the logistic sigmoid and \(a\) a logit, \(d=\sigma(a)\).
Printings. The first printing has a sign error in (17.17), which makes the "minimum" of Exercise 17.1 a maximum of the error as defined in (17.16); a garbled gradient-penalty error (17.11); and a misleading description of mode collapse. Each is flagged where it occurs, and the list with the evidence is in the summary before the exercises.
p. 533 A generative model learns a distribution \(p(\bx\given\bw)\) from training data and then produces new samples from it; a conditional generative model \(p(\bx\given\bc,\bw)\) does so given a label or other conditioning vector \(\bc\) (a cat rather than a dog). Autoregressive language models (Section 12.3) are one family. This chapter starts the second family: a nonlinear latent-variable model, in which a simple random vector \(\bz\) is pushed through a deep network. GANs are the first of four such models in the book; variational autoencoders, normalizing flows and diffusion models (Chapters 18 to 20) are the others, and the companion course's advanced module treats them in its own way.
p. 534 Take a latent distribution
\[ p(\bz)=\N(\bz\given\bzero,\bI) \tag{17.1} \]and a generator network \(\bx=\bg(\bz,\bw)\). Together they define a distribution \(\pG\) of \(\bx\) implicitly: we can sample from it (draw \(\bz\), apply \(\bg\)) but cannot evaluate its density, so the parameters \(\bw\) cannot be fitted by maximum likelihood. The idea of generative adversarial networks (Goodfellow et al., 2014) is to train a second network, the discriminator, to tell generated samples from data, and to use its output as the training signal for the generator (the book's Figure 17.1, a diagram of the two networks, is not reproduced).
The book says the likelihood "cannot, in general, be evaluated in closed form". In fact it often does not exist at all.
Equal dimensions, invertible generator. If \(\bz\) and \(\bx\) both live in \(\R^D\) and \(\bg(\cdot,\bw)\) is a smooth bijection with a smooth inverse, the change-of-variables formula (Section 2.4) gives \(\pG(\bx)\)
Fewer latent than data dimensions. In image GANs \(\bz\in\R^M\) with \(M\) (about 100) far below \(D\) (thousands of pixels). Suppose \(\bg\) is Lipschitz with constant \(L\) (true for networks with Lipschitz activations and finite weights). Cover the cube \([-r,r]^M\) by \(n^M\) small cubes of side \(2r/n\); each has diameter \(2r\sqrt M/n\), so its image lies in a ball of radius \(2rL\sqrt M/n\), of volume \(c_D(2rL\sqrt M/n)^D\). The image of the big cube therefore has volume at most \(n^M c_D(2rL\sqrt M)^Dn^{-D}\to0\) as \(n\to\infty\), because \(M<D\). Taking \(r=1,2,\dots\) shows that \(\bg(\R^M)\) has Lebesgue measure zero in \(\R^D\): \(\pG\) is concentrated on a thin set and has no density. A data point off that set has likelihood zero, and a log likelihood of \(-\infty\) gives no gradient.
What a GAN needs instead. Only samples \(\bg(\bz,\bw)\) and the derivatives \(\partial\bg/\partial\bw\), both available from any network. The thin support has a price, which returns in Result 17.8: two such distributions almost never overlap, and divergences that compare densities point by point then stop being informative. ∎
See also the companion course, logistic regression and the cross-entropy error (Section 4.3.2), and in this course Section 6.4.2 (binary classification error).
p. 535 Label real data and generated ("synthetic") data by a binary target,
\[ t=1,\quad\text{real data}, \tag{17.2} \] \[ t=0,\quad\text{synthetic data}. \tag{17.3} \]The discriminator has one logistic-sigmoid output, read as the probability that its input is real,
\[ P(t=1)=d(\bx,\bphi), \tag{17.4} \]and is trained with the cross-entropy error
\[ E(\bw,\bphi)=-\frac1N\sum_{n=1}^N\{t_n\ln d_n+(1-t_n)\ln(1-d_n)\}, \tag{17.5} \]where \(d_n=d(\bx_n,\bphi)\). The inputs are real examples \(\bx_n\) and synthetic ones \(\bg(\bz_n,\bw)\) with \(\bz_n\sim p(\bz)\); splitting the sum by label,
\[ E_{\rm GAN}(\bw,\bphi)=-\frac1{N_{\rm real}}\sum_{n\in\rm real}\ln d(\bx_n,\bphi)-\frac1{N_{\rm synth}}\sum_{n\in\rm synth}\ln\big(1-d(\bg(\bz_n,\bw),\bphi)\big). \tag{17.6} \]The discriminator minimises this error and the generator maximises it, by gradient steps of opposite sign on a mini-batch,
\[ \Delta\bphi=-\lambda\nabla_\bphi E_n(\bw,\bphi), \tag{17.7} \] \[ \Delta\bw=\lambda\nabla_\bw E_n(\bw,\bphi), \tag{17.8} \]p. 536 usually alternating one step of each and drawing new synthetic samples every time. When training is over, the discriminator is discarded and the generator samples by mapping fresh \(\bz\).
Let \(p=\pd\) be the data density and \(q=\pG\) the generator's. With infinitely many samples, (17.6) becomes
\[ E(q,d)=-\int p(\bx)\ln d(\bx)\,\d\bx-\int q(\bx)\ln\big(1-d(\bx)\big)\,\d\bx . \tag{17.14} \]The discriminator chooses a function \(d:\R^D\to(0,1)\) to minimise \(E\); the generator chooses \(q\) (through \(\bw\)) to maximise it. Since one player's gain is the other's loss, this is a two-player zero-sum game. Goodfellow et al. (2014) write it with the opposite sign, as \(\min_G\max_D V\) with \(V=-E\); the two descriptions are the same game.
Normalisation. With \(N=N_{\rm real}+N_{\rm synth}\), (17.5) is \(-\frac1N\sum_{\rm real}\ln d_n-\frac1N\sum_{\rm synth}\ln(1-d_n)\) because \(t_n\) kills one of the two terms. (17.6) normalises each group by its own size; for \(N_{\rm real}=N_{\rm synth}=N/2\) it is exactly twice (17.5), which changes no minimiser. With unequal group sizes (17.5) weights the two classes by \(\pi=N_{\rm real}/N\) and \(1-\pi\); (17.6) weights them equally, which is what makes \(d^\star\) in Result 17.1 independent of the batch composition.
Population limit. Each average in (17.6) is a Monte Carlo estimate (Section 14.1.1): \(\frac1{N_{\rm real}}\sum\ln d(\bx_n)\)
Gradients. \(\nabla_\bphi\) acts on \(d\) only, so the mini-batch gradient is an unbiased estimate of \(\nabla_\bphi E\). For \(\bw\) the expectation is over \(q\), which depends on \(\bw\); writing it over \(\bz\) instead, \(\E_q[\ln(1-d(\bx))]\)
This is the reparameterisation estimator of Filling the gap 14.2: the generator learns only through the input gradient \(\nabla_\bx d\) of the discriminator at its samples, a fact that drives everything in Section 17.1.2. In code, backpropagation runs through the discriminator into the generator; the discriminator's weights are simply not updated in that step.
Simultaneous and alternating steps. (17.7)–(17.8) taken together are gradient descent–ascent. "Simultaneous" evaluates both gradients at the same \((\bw,\bphi)\); "alternating" (the book's recipe, and the demos') updates \(\bphi\) first and then evaluates \(\nabla_\bw\) at the new \(\bphi\). Result 17.3 shows that the difference matters. ∎
p. 536 If the generator becomes perfect the discriminator cannot do better than guessing and outputs 1/2 everywhere. The book states that with unlimited flexibility a fully optimised GAN reproduces the data distribution exactly (Exercise 17.1); the next two results prove it.
Let \(p\) and \(q\) be densities (with respect to a common measure). For fixed \(q\), the discriminator that minimises (17.14) over all functions \(d\) is
\[ d^\star(\bx)=\frac{\pd(\bx)}{\pd(\bx)+\pG(\bx)} \tag{17.15} \]on the set where \(p+q>0\) (its value elsewhere does not matter). It is the posterior probability that \(\bx\) is real when real and synthetic inputs are equally likely a priori.
The integrand of (17.14) at a point \(\bx\) depends only on the number \(y=d(\bx)\): it is \(\varphi(y)=-a\ln y-b\ln(1-y)\) with \(a=p(\bx)\ge0\), \(b=q(\bx)\ge0\). If \(a,b>0\), \(\varphi\) is strictly convex on \((0,1)\), since \(\varphi''(y)=a/y^2+b/(1-y)^2>0\), tends to \(+\infty\) at both ends, and \(\varphi'(y)=-a/y+b/(1-y)\) vanishes only at \(y=a/(a+b)\). If \(b=0<a\), \(\varphi\) decreases towards \(y=1\) (and \(d^\star=1\) is the limit); if \(a=0<b\), towards \(y=0\). So \(\varphi(d(\bx))\)
A flexible network trained on samples approaches \(d^\star\) closely: Figure 17.S1(a) overlays a 1-32-32-1 network trained by cross-entropy on 20 000 samples of each kind.
For distributions \(P,Q\) with mixture \(M=\frac12(P+Q)\),
\[ \JS(P,Q)=\tfrac12\KL(P\|M)+\tfrac12\KL(Q\|M). \]This is the standard definition (Lin, 1991). The exercise calls the sum \(\KL(P\|M)+\KL(Q\|M)\) the Jensen–Shannon divergence, which is twice this; the factor matters only for the bound \(\JS\le\ln2\).
(1) Substituting (17.15) into (17.14) gives (17.16). Write \(\frac{p}{p+q}=\frac p{2M}\): then \(-\int p\ln\frac p{2M}\)
(2) Gibbs' inequality (Result 2.10; companion Result 1.22) gives \(\KL\ge0\) with equality iff the two arguments agree, so \(\JS\ge0\), with equality iff \(P=M=Q\). For the upper bound, \(\KL(P\|M)\)
(3) Follows from (1) and (2). ∎
"A fully optimised GAN matches the data" is true in function space with the discriminator optimised first. Three things separate it from practice.
A weak equilibrium. \(E(q,d)\) is convex in \(d\) (pointwise, as in the proof of Result 17.1) and linear in \(q\). The pair \((q,d)=(p,\frac12)\) is a saddle point: \(E(p,d)\ge E(p,\frac12)=\ln4\) for every \(d\) (because \(\frac12=d^\star\) when \(q=p\)), and \(E(q,\frac12)\)
Parameters, not functions. Networks reach only some \(q\) and \(d\). If no generator equals \(\pd\), the best one depends on the divergence: in Figure 17.S1(b) the JS-optimal Gaussian sits between the two bumps. And in \((\bw,\bphi)\) the error is neither convex nor concave, so the saddle point need not attract gradient play.
Gradient play can circle. Near an equilibrium, gradient descent–ascent behaves like its linearisation, the vector field \(F(\bw,\bphi)\)
Let \(E(a,b)=ab\); \(a\) (the "generator") ascends and \(b\) (the "discriminator") descends, with step or rate \(\eta\).
(1) \(\nabla E=(b,a)\) vanishes only at the origin; the Hessian \(\begin{pmatrix}0&1\\1&0\end{pmatrix}\) has eigenvalues \(\pm1\) with eigenvectors along \(b=\pm a\); along \(b=a\), \(E=a^2\), with second derivative \(2\), and along \(b=-a\), \(E=-a^2\), with second derivative \(-2\).
(2) is Exercise 17.2, solved below.
(3)–(5) Write \(\bz=(a,b)^\T\) and \(J=\begin{pmatrix}0&1\\-1&0\end{pmatrix}\), so the update direction is \(J\bz\) and \(J^\T J=\bI\), \(J^\T=-J\), \(J^2=-\bI\). Simultaneous: \(\|(\bI+\eta J)\bz\|^2\)
Alternating updates are thus a real improvement over simultaneous ones (bounded instead of divergent), but neither converges; damping (momentum with a negative coefficient, extragradient, optimistic steps, or a regulariser such as the gradient penalty) is needed for convergence (Mescheder, Geiger and Nowozin, 2018).
Conditional GANs. p. 536 To sample from \(p(\bx\given\bc)\) (a chosen class, say), both networks receive \(\bc\) as an extra input and are trained on labelled pairs \((\bx_n,\bc_n)\) (Mirza and Osindero, 2014). One network for all classes shares its internal representations across them, which uses the data more efficiently than one GAN per class.
Let labels \(\bc\) have distribution \(p(\bc)\), data \(p(\bx\given\bc)\), and let the generator produce \(\pG(\bx\given\bc)\) for labels drawn from the same \(p(\bc)\). The optimal discriminator is \(d^\star(\bx,\bc)\)
maximised exactly when \(\pG(\cdot\given\bc)=p(\cdot\given\bc)\) for every \(\bc\) with \(p(\bc)>0\).
Apply Result 17.1 to the joint densities \(p(\bc)p(\bx\given\bc)\) and \(p(\bc)\pG(\bx\given\bc)\) of the pair \((\bx,\bc)\): the common factor \(p(\bc)\) cancels in \(d^\star\). In Result 17.2 the mixture is \(p(\bc)M(\bx\given\bc)\) with \(M=\frac12(p+\pG)\) conditionally, and \(\KL(p(\bc)p(\cdot\given\bc)\,\|\,p(\bc)M(\cdot\given\bc))\)
p. 536 GANs are hard to train because of the adversarial set-up, and the error can go up as well as down, so it does not measure progress (Exercise 17.2 and Figure 17.S5(c) below). One common failure is mode collapse: the generator maps all latent vectors to a small set of outputs, for instance only one digit. The book describes the discriminator as then assigning 1/2 to these outputs, after which training stops. That is not what the optimal discriminator does, and mode collapse is better understood as a best response.
(1) \(\E_q[\ell]=\int q\ell\) with \(\ell=\ln(1-d)\) (to be minimised) or \(-\ln d\); both are decreasing functions of \(d\). For any \(q\), \(\int q\ell\ge\inf_\bx\ell(\bx)\), with equality for \(q=\delta_{\bx_0}\) at a minimiser of \(\ell\), i.e. a maximiser of \(d\). A generator that is improved against the current discriminator is therefore pulled towards concentrating its mass where \(d\) is largest. (2) By Result 17.1 applied to the sets \(A\) of the mode: \(\int_A\pd=\pi\), \(\int_A\pG=1\), and \(d^\star=\pd/(\pd+\pG)\) is small wherever \(\pG\) is large.
What actually happens. Real training sits between these two extremes: the discriminator takes a step or a few per generator step. The generator piles onto the modes the current \(d\) favours; the discriminator then lowers \(d\) there; the pile moves on. The result is mode hopping rather than a frozen state: in Figure 17.S5(a) the number of covered modes of the run with a slow discriminator swings between 2 and 8. Training "ceases" only if the discriminator cannot represent the difference, for example if it never sees enough samples at once to notice that a mode is missing (minibatch discrimination, Salimans et al., 2016, gives it that view). ∎
p. 537 Figure 17.2 shows why training can be slow. When data and generator distributions are far apart, the optimal discriminator is easy to learn and falls off very steeply, with almost zero gradient near both sets of samples. In the second term of (17.6), \(d(\bg(\bz,\bw),\bphi)\) is then essentially zero across the generated samples, so small changes of \(\bw\) barely change the error.
For \(p=\N(m_0,s^2)\) and \(q=\N(m_1,s^2)\), \(d^\star=\sigma(a)\) with logit \(a(x)\)
p. 537 A smoothed discriminator \(\widetilde d\) gives the generator a stronger signal. The least-squares GAN (Mao et al., 2016) gets it with a real-valued discriminator trained by a sum-of-squares error; instance noise (Sønderby et al., 2016) adds Gaussian noise to both real and synthetic samples.
Why the gradient vanishes in general, why noise brings it back, and why noise does not change the answer.
Vanishing. By Result 17.7(c), the minimax gradient against \(d^\star\) is \(\nabla_\btheta\,2\JS(\pd,\pG)\). When the two distributions do not overlap, JS is constant (\(\ln2\)) under any small change that keeps them apart (Result 17.8), so the gradient is exactly zero; when they overlap only in far tails it is exponentially small. For \(\pd=\N(0,0.3^2)\) and \(\pG=\N(4,0.3^2)\) the grid computation gives \(9\times10^{-10}\) (minimax) against 44 (non-saturating).
Noise. Replace every real and synthetic input \(\bx\) by \(\bx+\bm\epsilon\), \(\bm\epsilon\sim\N(\bzero,\tau^2\bI)\). The distributions become the convolutions \(p_\tau=p*\N_\tau\) and \(q_\tau=q*\N_\tau\), both positive everywhere and smooth; the optimal discriminator \(p_\tau/(p_\tau+q_\tau)\) is then smooth, and \(\JS(p_\tau,q_\tau)<\ln2\) responds to every movement of \(q\). For Gaussian mixtures convolution only adds \(\tau^2\) to each variance (confirmed numerically on the grid); with \(\tau=1\) the gradient in the example above becomes 0.15.
The optimum is unchanged. The characteristic function of \(\N(\bzero,\tau^2\bI)\), \(e^{-\tau^2\|\bm\omega\|^2/2}\), never vanishes, and convolution multiplies characteristic functions, so \(p_\tau=q_\tau\) implies \(p=q\). Minimising \(\JS(p_\tau,q_\tau)\) therefore still has \(\pG=\pd\) as its unique solution; in practice \(\tau\) is reduced during training. ∎
Let the discriminator minimise \(\frac12\E_p[(d-b)^2]+\frac12\E_q[(d-a)^2]\) (target \(b\) for real, \(a\) for synthetic inputs) and the generator minimise \(\frac12\E_p[(d-c)^2]+\frac12\E_q[(d-c)^2]\) (the first term does not depend on the generator).
(1) Pointwise, \(\frac12p(y-b)^2+\frac12q(y-a)^2\) is a convex quadratic in \(y=d(\bx)\) with minimum where \(p(y-b)+q(y-a)=0\). (2) \(d^\star_{\rm LS}-c\)
Part (1) shows that, at the optimum, the least-squares discriminator is exactly as steep as the GAN one: the smoothing of Figure 17.2 cannot come from the optimum. It comes from what a discriminator of limited capacity does during training (next box), and from the loss \((d-c)^2\), whose gradient does not vanish when \(d\) is far from \(c\).
Take the simplest discriminator, linear in \(x\), and real and synthetic samples that a threshold separates. Cross-entropy, \[ -\sum_n\big[t_n\ln\sigma(wx_n+b)+(1-t_n)\ln(1-\sigma(wx_n+b))\big], \] has no minimiser: scaling a separating \((w,b)\) by \(k>1\) lowers every term, so gradient descent drives \(|w|\to\infty\), the sigmoid becomes a step, and its gradient vanishes at the samples. On separable data the norm grows only logarithmically in the number of steps while the direction converges to the maximum-margin separator (Soudry et al., 2018); Figure 17.S6(b) shows \(|w|\) gaining the same amount per decade of steps. Least squares has the unique minimiser given by the normal equations (a finite slope, 0.238 here), and the generator's loss \((wx+b-1)^2\) has gradient \(2(wx+b-1)w\), which does not decay when the samples are rejected. Panel (c) runs a one-dimensional GAN (data \(\N(0,1)\), generator \(x=\mu+z\) started at \(\mu=6\), five discriminator steps per generator step): after 300 generator steps the minimax generator has moved to 5.18, the least-squares one to 1.00, and the non-saturating one has arrived (\(-0.02\)). ∎
pp. 537–538 The most common change (Mescheder, Geiger and Nowozin, 2018, review many others) replaces the generator's term
\[ -\frac1{N_{\rm synth}}\sum_{n\in\rm synth}\ln\big(1-d(\bg(\bz_n,\bw),\bphi)\big) \tag{17.9} \]by
\[ \frac1{N_{\rm synth}}\sum_{n\in\rm synth}\ln d(\bg(\bz_n,\bw),\bphi), \tag{17.10} \]still to be maximised by the generator: instead of making a sample less likely to be called fake, it makes it more likely to be called real. Figure 17.3 compares the two.
(1) \(\sigma'=\sigma(1-\sigma)\), so \(\partial_a\ln(1-\sigma)\)
(2) At \(q=p\), \(d^\star=\frac12\), so both \(\E_q[\ln(1-d^\star)]\) and \(\E_q[-\ln d^\star]\) are \(\int q\cdot\ln\frac12\) and \(\int\delta q\cdot\ln\frac12=0\) for any perturbation with \(\int\delta q=0\).
(3) Write \(q=q_\btheta\), \(\dot q=\partial q/\partial\theta_i\), and assume differentiation under the integral is allowed. From (17.16) and Result 17.2, \(2\JS(p,q)-\ln4\)
Consequence: mode dropping. \(\KL(q\|p)\) is huge where \(q\) puts mass and \(p\) has none, but costs little when \(q\) ignores part of \(p\) (the mode-seeking property of the reverse KL; companion Section 10.1.2). On two bumps the non-saturating objective is minimised by a Gaussian on one bump (Figure 17.S1(c)), and descent against \(d^\star\) ends there: \((\mu,\sigma)=(2,0.5)\), while the minimax descent ends at the covering Gaussian \((0,2.04)\). The non-saturating loss buys strong gradients at the price of this bias, which is consistent with the far-start ring run of Figure 17.S4 arriving fast but collapsing onto four modes.
p. 538 A more direct remedy is to measure how far apart \(\pG\) and \(\pd\) are in data space. The Wasserstein or earth mover's distance pictures \(\pG\) as a pile of earth to be reshaped into \(\pd\): it is the least possible amount of earth times average distance moved. It cannot be computed directly; the Wasserstein GAN (Arjovsky, Chintala and Bottou, 2017) approximates it with a real-valued discriminator (a critic) whose input gradient \(\nabla_\bx d(\bx,\bphi)\) is kept bounded by weight clipping, and the gradient-penalty version (Gulrajani et al., 2017) replaces the clipping by a penalty. First, why JS is the wrong yardstick when supports do not overlap.
If \(P\) and \(Q\) are concentrated on disjoint sets \(A\) and \(B\), then \(\JS(P,Q)=\ln2\), \(\KL(P\|Q)=\KL(Q\|P)=\infty\), the indicator of \(A\) is an optimal discriminator with error 0, and for any family \(Q_\btheta\) that stays disjoint from \(A\) near \(\btheta_0\), \(\nabla_\btheta\JS(P,Q_\btheta)=\bzero\) there. When \(\pG\) is supported on the image of a lower-dimensional latent space (Filling the gap 17.1) and \(\pd\) is concentrated near a low-dimensional data manifold, the two supports generically do not overlap, so this is the typical situation early in training (Arjovsky and Bottou, 2017).
On \(A\), \(q=0\) and \(M=p/2\), so \(\KL(P\|M)=\int_Ap\ln2=\ln2\); likewise \(\KL(Q\|M)=\ln2\); hence \(\JS=\ln2\) whatever the distance between \(A\) and \(B\). \(\KL(P\|Q)=\int p\ln(p/q)\) has \(q=0\) where \(p>0\). The indicator of \(A\) gives \(-\ln1\) on real and \(-\ln(1-0)\) on synthetic samples, i.e. zero error. A constant function of \(\btheta\) has zero gradient. ∎ For \(P=U[0,1]\) and \(Q=U[\theta,\theta+1]\): \(\JS=|\theta|\ln2\) for \(|\theta|\le1\) and \(\ln2\) beyond, while the earth mover's distance is \(|\theta|\) (Figure 17.S7(a)).
For distributions \(P,Q\) on \(\R^D\) with finite means, a coupling is a joint distribution \(\gamma\) of \((\bx,\by)\) with marginals \(P\) and \(Q\) (a transport plan: \(\gamma(\bx,\by)\) is the amount of earth moved from \(\by\) to \(\bx\)). The Wasserstein-1 distance is the cheapest plan's average distance,
\[ \W(P,Q)=\inf_{\gamma\in\Pi(P,Q)}\E_{(\bx,\by)\sim\gamma}\|\bx-\by\| . \]For distributions on \(\R\) with distribution functions \(F_P,F_Q\) and quantile functions \(F_P^{-1},F_Q^{-1}\),
\[ \W(P,Q)=\int_{-\infty}^{\infty}|F_P(x)-F_Q(x)|\,\d x=\int_0^1|F_P^{-1}(u)-F_Q^{-1}(u)|\,\d u=\sup_{\Lip(f)\le1}\big(\E_P[f]-\E_Q[f]\big), \]the infimum is attained by the quantile coupling \((F_P^{-1}(U),F_Q^{-1}(U))\), \(U\sim U(0,1)\), and the supremum by the optimal critic \(f^\star\) with \(f^{\star\prime}=\sgn(F_Q-F_P)\). In any dimension, the equality of the infimum and the supremum over 1-Lipschitz functions (Kantorovich–Rubinstein duality) still holds (Villani, 2009, Chapter 5); the WGAN critic is a network that approximates the supremum.
Weak duality (any dimension). For a coupling \(\gamma\) and \(\Lip(f)\le1\), \(\E_P f-\E_Q f\)
The supremum equals the area between the distribution functions. A 1-Lipschitz \(f\) is absolutely continuous with \(|f'|\le1\). Integrating by parts (for finite means the boundary terms vanish), \[ \E_P f=f(0)+\int_0^\infty f'(x)(1-F_P(x))\,\d x-\int_{-\infty}^0f'(x)F_P(x)\,\d x ; \] subtracting the same expression for \(Q\),
\[ \E_P f-\E_Q f=\int_{-\infty}^\infty f'(x)\big(F_Q(x)-F_P(x)\big)\,\d x\le\int|F_Q-F_P|\,\d x , \]with equality for \(f'=\sgn(F_Q-F_P)\).
The quantile coupling costs the same area. Its cost \(\int_0^1|F_P^{-1}(u)-F_Q^{-1}(u)|\,\d u\) is the area of the region between the graphs of \(F_P\) and \(F_Q\) measured in horizontal strips, and \(\int|F_P-F_Q|\,\d x\) is the same area measured in vertical strips (Fubini). So \(\inf\le\int|F_P-F_Q|=\sup\le\inf\), and all are equal. ∎
Numerically: for ten random pairs of mixtures the two integrals agree to \(2\times10^{-3}\) on the grid and match the optimal transport of \(2\times10^5\) sorted samples; \(\E_Pf^\star-\E_Qf^\star=\W\) to \(10^{-6}\), and 200 random 1-Lipschitz functions all give less. For a translation, \(F_Q^{-1}(u)=F_P^{-1}(u)+\theta\), so \(\W=|\theta|\).
If \(\|\bg(\bz,\btheta)-\bg(\bz,\btheta')\|\)
\(\W\) is a metric (it is symmetric, vanishes only for equal distributions, and obeys the triangle inequality by gluing an optimal plan from \(P\) to \(Q\) with one from \(Q\) to \(R\); Villani, 2009, Chapter 6). Hence \(|\W(\pd,P_\btheta)-\W(\pd,P_{\btheta'})|\)
p. 538 The gradient-penalty WGAN is written in the book as
\[ E_{\rm WGAN\text{-}GP}(\bw,\bphi)=-\frac1{N_{\rm real}}\sum_{n\in\rm real}\Big[\ln d(\bx_n,\bphi)-\eta\big(\|\nabla_{\bx_n}d(\bx_n,\bphi)\|^2-1\big)^2\Big]+\frac1{N_{\rm synth}}\sum_{n\in\rm synth}\ln d(\bg(\bz_n,\bw,\bphi)) \tag{17.11} \]where \(\eta\) weights the penalty. This differs from the method it cites in four ways (see the next box and the errata); the version that Gulrajani et al. (2017) actually use, and that the demo and code implement, is
\[ E_{\rm critic}(\bphi)=\frac1{N}\sum_{n}d(\bg(\bz_n,\bw),\bphi)-\frac1N\sum_nd(\bx_n,\bphi)+\frac\eta N\sum_n\big(\|\nabla_\bx d(\widehat\bx_n,\bphi)\|-1\big)^2,\qquad \widehat\bx_n=\epsilon_n\bx_n+(1-\epsilon_n)\bg(\bz_n,\bw), \]with \(\epsilon_n\sim U(0,1)\), minimised over \(\bphi\), while the generator minimises \(-\frac1N\sum_nd(\bg(\bz_n,\bw),\bphi)\).
Lipschitz means bounded gradient. For differentiable \(f\), \(\Lip(f)\le1\) if and only if \(\|\nabla f(\bx)\|\le1\) everywhere: by the mean value theorem \(|f(\bx)-f(\by)|\)
Weight clipping keeps every weight in \([-c,c]\). For a network with 1-Lipschitz activations, \(\Lip(f)\)
Why penalise at interpolates. Result 17.11 shows that an optimal critic has gradient norm exactly 1 on the segments between points that the optimal plan couples. Real and generated samples paired at random are a cheap stand-in for coupled pairs, and points \(\widehat\bx\) on the segments between them are where the constraint is active. Penalising at the real points only, as (17.11) is written, constrains the critic where its gradient matters least.
The penalty is a soft constraint. For \(P=U[0,1]\), \(Q=U[3,4]\) and linear critics \(f(x)=sx\), the objective \(\E_Pf-\E_Qf-\lambda(|s|-1)^2\)
Let \(f^\star\) attain the supremum in Result 17.9 and \(\gamma^\star\) the infimum. Then \(f^\star(\bx)-f^\star(\by)=\|\bx-\by\|\) for \(\gamma^\star\)-almost every pair, \(f^\star\) is linear with slope 1 on the segment from \(\by\) to \(\bx\), and wherever \(f^\star\) is differentiable on that segment, \(\nabla f^\star=(\bx-\by)/\|\bx-\by\|\).
By duality, \(\E_{\gamma^\star}[f^\star(\bx)-f^\star(\by)]\)
See also this course, Section 10.5.2 (up-sampling and transposed convolutions, Result 10.9) and the convolution arithmetic demo.
p. 539 Images are the most successful application of GANs. Early GANs used fully connected networks, but convolutional ones work much better: the discriminator is an ordinary convolutional classifier, and the generator maps a low-dimensional \(\bz\) to a large image through transposed convolutions (the book's Figure 17.4, a DCGAN diagram from \(\bz\in\R^{100}\) through \(4\times4\times1024\), \(8\times8\times512\), \(16\times16\times256\) and \(32\times32\times128\) to a \(64\times64\times3\) image). Progressive growing of both networks, from \(4\times4\) up to \(1024\times1024\) images, speeds up training and allows very high resolution (Karras et al., 2017); BigGAN (Brock, Donahue and Simonyan, 2018; the book's Figure 17.5) is a class-conditional generator with residual blocks and about 70 million parameters. Neither diagram is reproduced.
A transposed convolution with kernel \(k\), stride \(s\) and padding \(p\) maps width \(i\) to \(o=s(i-1)+k-2p\) (it is the adjoint of the strided convolution mapping \(o\) to \(i\); Result 10.9). With \(k=4\), \(s=2\), \(p=1\), \(o=2i\), giving \(4\to8\to16\to32\to64\). PyTorch confirms the sizes and the adjoint identity \(\langle\mathrm{conv}(\bx),\by\rangle\)
p. 542 GANs also learn representations (Section 6.3.3). A DCGAN trained on bedroom images (Radford, Metz and Chintala, 2015) produces bedrooms, and its latent space is organised: a smooth path in \(\bz\) gives a smooth sequence of plausible images (the book's Figure 17.9), and some directions correspond to meaningful changes such as pose, lighting or a smile, disentangled representations that support arithmetic on latent vectors (Figure 17.10). Below, both phenomena are reproduced with a small GAN on MNIST.
Norms concentrate. For \(\bz\sim\N(\bzero,\bI_D)\), \(\|\bz\|^2\) is \(\chi^2_D\), with mean \(D\) and variance \(2D\), so \(\|\bz\|/\sqrt D\to1\): in \(D=32\) almost all latent vectors lie near the sphere of radius \(\sqrt{32}\approx5.7\), and the generator has been trained only there. The midpoint of a straight line between two independent latents, \((\bz_1+\bz_2)/2\sim\N(\bzero,\frac12\bI)\), has \(\E\|\cdot\|^2=D/2\): it lies well inside the sphere, in a region the generator never saw (a simulation gives \(0.500D\)). Spherical interpolation \(\mathrm{slerp}(\bz_1,\bz_2;t)\)
Averaging and arithmetic. The mean of \(k\) independent latents has covariance \(\bI/k\), so it is shorter than a typical latent by \(\sqrt k\). The arithmetic of Figure 17.10, \(\bar\bz_A-\bar\bz_B+\bar\bz_C\) with three vectors per group, has covariance \(3\cdot\bI/3=\bI\) if the nine vectors are independent: the result is again a typical latent vector, while the averages remove much of the individual variation. ∎
Why does pixel arithmetic fail where latent arithmetic works? An average of images is an image only if the images are aligned; strokes at slightly different places superpose as ghosts (the bottom right of Figure 17.10, and the bottom row of the book's Figure 17.10). The generator maps latent vectors back onto (an approximation of) the set of digit images, so arithmetic done before the generator ends on that set.
pp. 539–542 CycleGAN (Zhu et al., 2017) learns to translate between two domains, for example photographs \(X\) and Monet paintings \(Y\), from two unpaired collections (the book's Figure 17.6 shows translations in both directions). It has two conditional generators, \(\bg_X(\by,\bw_X)\) turning a painting into a photograph and \(\bg_Y(\bx,\bw_Y)\) turning a photograph into a painting, and two discriminators \(d_X\) (real or synthetic photograph?) and \(d_Y\) (real or synthetic painting?). The usual GAN errors alone would make the outputs look like the right domain, but nothing would tie a painting to the photograph it came from. CycleGAN adds the cycle-consistency error (the book's Figure 17.7): a photograph translated to a painting and back should return to itself, and likewise for paintings,
\[ E_{\rm cyc}(\bw_X,\bw_Y)=\frac1{N_X}\sum_{n\in X}\|\bg_X(\bg_Y(\bx_n))-\bx_n\|_1+\frac1{N_Y}\sum_{n\in Y}\|\bg_Y(\bg_X(\by_n))-\by_n\|_1 , \tag{17.12} \]with \(\|\cdot\|_1\) the L1 norm, and the total error (whose information flow is the book's Figure 17.8, a diagram not reproduced) is
\[ E_{\rm GAN}(\bw_X,\bphi_X)+E_{\rm GAN}(\bw_Y,\bphi_Y)+\eta E_{\rm cyc}(\bw_X,\bw_Y) . \tag{17.13} \](1) The GAN error of \(\bg_Y\) depends on \(\bg_Y\) only through the distribution of \(\bg_Y(\bx)\), which is unchanged by \(T\). (2) A non-negative integrand with zero integral vanishes almost everywhere; \(\bg_X\circ\bg_Y=\mathrm{id}\) on a set implies \(\bg_Y\) is injective there (equal images would give equal pre-images). If also \(\bg_Y\circ\bg_X=\mathrm{id}\) on the support of \(p_Y\), each map is onto the other's support up to null sets, so they are mutually inverse bijections. (3) Any bijection \(\bg_Y\) pushing \(p_X\) to \(p_Y\), with \(\bg_X=\bg_Y^{-1}\), makes all three errors optimal. ∎
The book does not discuss how to tell whether a GAN is good, but the question is unavoidable: its loss is no guide (Figure 17.S5(c)) and it has no likelihood (Filling the gap 17.1). Practice compares statistics of generated and real samples in the feature space of a pretrained classifier. Below the classifier is a small MNIST network (95.0% test accuracy) and its 256 hidden units are the features.
With a classifier \(p(y\given\bx)\) over \(K\) classes and \(p(y)=\E_{\bx\sim\pG}p(y\given\bx)\), the inception score (Salimans et al., 2016) is \(\mathrm{IS}\)
Fit Gaussians \(\N(\bmu_1,\Sig_1)\) and \(\N(\bmu_2,\Sig_2)\) to the features of real and generated images. Then
\[ \mathrm{FD}=\|\bmu_1-\bmu_2\|^2+\Tr\Big(\Sig_1+\Sig_2-2\big(\Sig_1^{1/2}\Sig_2\Sig_1^{1/2}\big)^{1/2}\Big) \]is the squared Wasserstein-2 distance \(\inf_\gamma\E_\gamma\|\bx-\by\|^2\) between the two Gaussians (Dowson and Landau, 1982); computed in the features of an Inception network it is the FID.
Centre both (the means contribute \(\|\bmu_1-\bmu_2\|^2\) to any coupling's cost, since the cross term has mean zero). Let \(T\)
On the ring the same questions have exact answers: the number of modes with nearby samples measures diversity (recall) and the fraction of samples near a mode measures quality (precision), which is how Figures 17.S4–17.S5 and the ring demo report progress.
All three exercises of Chapter 17 (Bishop and Bishop, 2024, pp. 544–545), restated in my own words, with complete solutions (hidden). Stars give the book's difficulty. The official solutions manual stops at Chapter 10, so these solutions are my own.
Exercise 17.1 (★★★). Prove that, with unlimited flexibility, the GAN's solution is the data distribution. Optimise over all distributions \(\pG\) and all functions \(d\), with \(d\) optimised in an inner loop. (a) Show that in the limit of infinitely many samples (17.6) becomes (17.14). (b) Show by a variational argument that the optimal \(d\) for fixed \(\pG\) is (17.15). (c) Show that the error then becomes (17.16), (d) rewrite it with Kullback–Leibler divergences as in (17.17), and (e) use \(\KL\ge0\), with equality only for equal arguments, to show that the optimum is \(\pG=\pd\). The two KL terms together form the Jensen–Shannon divergence, which, unlike KL, is symmetric.
(a) The real \(\bx_n\) are independent draws from \(\pd\), and the \(\bg(\bz_n,\bw)\), \(\bz_n\sim p(\bz)\), independent draws from \(\pG\). By the strong law of large numbers, \(\frac1{N_{\rm real}}\sum\ln d(\bx_n)\)
(b) Perturb \(d\) to \(d+\epsilon\eta\) for an arbitrary function \(\eta\): \(\frac{\d}{\d\epsilon}E\big|_{\epsilon=0}\)
(c) Substituting \(d^\star\) and \(1-d^\star=\pG/(\pd+\pG)\) into (17.14) gives (17.16).
(d) With \(M=(\pd+\pG)/2\), \(\frac{\pd}{\pd+\pG}=\frac{\pd}{2M}\), so \(-\int\pd\ln\frac{\pd}{2M}\)
(e) Both KL terms are \(\ge0\) and vanish iff \(\pd=M\) and \(\pG=M\), i.e. iff \(\pG=\pd\). So \(C(\pG)\le\ln4\) with equality iff \(\pG=\pd\): the generator, which maximises the error, attains its best value \(\ln4\) exactly at the data distribution, where \(d^\star\equiv\frac12\). In the printed form, \(-\ln4+\KL+\KL\ge-\ln4\) with the same equality case, which is the "minimum" of the exercise. The sum \(\KL(\pd\|M)+\KL(\pG\|M)\) is \(2\JS\) in the usual normalisation (Definition 17.2); it is symmetric because swapping \(\pd\) and \(\pG\) leaves \(M\) unchanged and swaps the two terms. Numerical confirmation: Result 17.2 and the demo, preset 2.
Exercise 17.2 (★★★). Let \(E(a,b)=ab\), with \(a\) playing the generator and \(b\) the discriminator. Show that \((0,0)\) is a stationary point, and, from the second derivatives along \(b=a\) and \(b=-a\), that it is a saddle. In continuous time, with \(a\) ascending and \(b\) descending, the parameters follow (17.18), \(\dot a=\eta\,\partial E/\partial a\), \(\dot b=-\eta\,\partial E/\partial b\). Show that \(a\) satisfies \(\ddot a=-\eta^2a\) (17.19), that \(a=C\cos\eta t+D\sin\eta t\) (17.20) solves it, find \(C,D\) for the start \(a=1\), \(b=0\), and show that \((a,b)\) then moves on the unit circle and never reaches the saddle.
Stationary point and saddle. \(\partial E/\partial a=b\) and \(\partial E/\partial b=a\) both vanish at the origin. Along \(b=a\), \(E=a^2\), whose second derivative in \(a\) is \(+2\) (a minimum along this line); along \(b=-a\), \(E=-a^2\), second derivative \(-2\) (a maximum). A point that is a minimum along one line and a maximum along another is a saddle; equivalently the Hessian \(\begin{pmatrix}0&1\\1&0\end{pmatrix}\) has eigenvalues \(\pm1\).
Dynamics. (17.18) reads \(\dot a=\eta b\), \(\dot b=-\eta a\). Differentiating the first and substituting the second, \(\ddot a=\eta\dot b=-\eta^2a\), which is (17.19). For (17.20), \(\dot a=\eta(-C\sin\eta t+D\cos\eta t)\) and \(\ddot a\)
Discrete steps do worse or the same: simultaneous gradient steps spiral outwards, alternating steps cycle, and only a corrected method such as extragradient converges (Result 17.3, Figure 17.S2). Integrating (17.18) numerically matches \(\cos\eta t\), \(-\sin\eta t\) to \(10^{-11}\).
Exercise 17.3 (★). The training set contains equal numbers of cat and dog images, and the generator has learned to produce high-quality dog images. Show that the optimal discriminator outputs \(1/3\) on a dog image.
Write \(\pd\)
and on a cat image \(d^\star=1\). Interpretation: a dog image comes from the data with prior 1/2, but the data produce dogs at half the generator's rate, so the posterior odds of "real" are \(\frac12:1\). The generator is not at the optimum: \(\JS\)
Three interactive pages, each linked above where its topic arises. They run in the browser and reproduce the Python code number for number (largest differences between page and Python \(8\times10^{-14}\) for 200 iterations of the ring GAN with each loss, \(10^{-13}\) for the optimal-discriminator quantities and descent paths, \(10^{-12}\) for the Wasserstein and JS curves). The companion course has no GAN demonstration; its VAE latent explorer, normalizing-flow and diffusion demos show the likelihood-based alternatives.
Every numerical statement in these notes comes from ch17_generative_adversarial_networks.py.