Interactive demo · Foundations of Deep Learning · Chapter 18

A continuous flow: the trace term and Hutchinson's estimator

A velocity field f(z, t) (a small tanh network trained in Python) moves points by dz/dt = f (18.27) from a standard Gaussian at t = 0 to the two moons at T = 1. Along each path the log density changes at the rate d ln p/dt = −Tr(∂f/∂z) (18.28): where the flow contracts (negative trace) density piles up. To find ln p(x) the page integrates backwards from x at time t to t = 0 and adds up the trace. The trace can be computed exactly, or estimated by Hutchinson's εT(∂f/∂z)ε (18.29)–(18.30) with a random probe ε held fixed along each path: unbiased, but noisy. Click the left panel to follow one point.

Time t

Runge–Kutta steps on [0, 1]

Trace of ∂f/∂z

Probes per point M

Try these

Velocity: f(z, t) = W3 tanh(W2 tanh(W1[z, t] + b1) + b2) + b3 with 32 units per layer, fitted by flow matching: f((1−t)z0 + tx1, t) was regressed on x1 − z0 (Lipman et al. 2022). Its Jacobian is exact: W3 diag(1−h2²) W2 diag(1−h1²) W1. The density map is computed on a 50 × 50 grid over [−3, 3]² each grid point is integrated back to t = 0 with the chosen number of steps (4 network evaluations per step). Probes are drawn from a seeded generator, Gaussian or Rademacher (±1), one set per grid point, fixed along the path. "Mass in the box" is the midpoint sum of the density.

In the lecture notes