Interactive demo · Foundations of Deep Learning · Chapter 14

Gradients of expectations: score function or reparameterisation?

To train through a sampling step we need ∂/∂θ of Ez~N(μ, σ²)[f(z)] with θ = μ or σ. The score-function estimator averages f(z) ∂ln p(z|θ)/∂θ and needs only values of f; the reparameterisation estimator writes z = μ + σε and averages f′(z) ∂z/∂θ. Both are unbiased when f is smooth, but their variances can differ by orders of magnitude, and either one can lose. Each histogram shows 2000 independent estimates, each from L samples (the same ε's for both).

Function f(z)

Differentiate with respect to

Mean μ

Standard deviation σ

Shift a

Samples per estimate L

Baseline for the score function

Try these

With ε ~ N(0, 1) and z = μ + σε: score function f(z)ε/σ (for μ) or f(z)(ε² − 1)/σ (for σ); reparameterisation f′(z) (for μ) or f′(z)ε (for σ). The baseline replaces f by f − E[f], which leaves the mean unchanged because E[∂ln p/∂θ] = 0. Exact means and variances are computed by quadrature over ε (trapezoid rule, 20001 points on [−10, 10]; for the step function the jump limits the accuracy to about 10−4).

In the lecture notes