Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 22

The masks of a MADE network

A masked autoencoder for density estimation (MADE) computes all \(D\) conditionals \(p(x_d\mid x_{\mathrm{before}\ d})\) in one pass of an ordinary network, by zeroing weights. Every input gets a position in the ordering and every hidden unit a degree; a weight is kept only if it cannot carry information from a later variable to an earlier one. The last panel multiplies the masks: entry \((d,d')\) counts the paths from input \(d'\) to output \(d\). It must vanish unless \(d'\) comes before \(d\).

Network

Ordering (position of each variable)

Rows and columns

Try these

Rule: hidden unit \(j\) gets degree \(m_j\in\{1,\dots,D-1\}\) (here \(1+(j \bmod (D-1))\)). Input \(d\) with position \(p_d\) may feed unit \(j\) iff \(m_j\ge p_d\); a hidden unit may feed a later hidden unit iff its degree is at least as large; unit \(j\) may feed output \(d\) iff \(p_d>m_j\). The first output (position 1) is connected to nothing and models the marginal of the first variable.

In the lecture notes