A masked autoencoder for density estimation (MADE) computes all \(D\) conditionals \(p(x_d\mid x_{\mathrm{before}\ d})\) in one pass of an ordinary network, by zeroing weights. Every input gets a position in the ordering and every hidden unit a degree; a weight is kept only if it cannot carry information from a later variable to an earlier one. The last panel multiplies the masks: entry \((d,d')\) counts the paths from input \(d'\) to output \(d\). It must vanish unless \(d'\) comes before \(d\).
Rule: hidden unit \(j\) gets degree \(m_j\in\{1,\dots,D-1\}\) (here \(1+(j \bmod (D-1))\)). Input \(d\) with position \(p_d\) may feed unit \(j\) iff \(m_j\ge p_d\); a hidden unit may feed a later hidden unit iff its degree is at least as large; unit \(j\) may feed output \(d\) iff \(p_d>m_j\). The first output (position 1) is connected to nothing and models the marginal of the first variable.
code/ch22_autoregressive_models.py (made_masks, path_matrix, Made, Figure x22.b).