One masked network (a MADE) reads a vector u and returns, for every i, a shift μi and a log-scale αi that depend only on u1, …, ui−1. Used as a masked autoregressive flow (MAF, (18.17)) it reads u = x and sets xi = μi + eαi zi; used as an inverse autoregressive flow (IAF, (18.19)) it reads u = z. The same formula, a different input, and the opposite cost: whichever direction has its input fully known takes one network pass; the other direction must fill the vector in one coordinate at a time, D sequential passes. Press Step to run one pass at a time.
Network: D inputs, H = max(8, 2D) tanh hidden units and 2D outputs. Hidden unit k has degree mk = 1 + (k mod (D − 1)) (the number in each grey node); input d feeds unit k only if mk ≥ d, and unit k feeds output i only if i > mk, so output i never sees inputs i, …, D (Germain et al. 2015). Only the connections that the masks keep are drawn. Weights are seeded normals (network seed shown in the readout), so this page and the Python code compute the same numbers. Jacobian: central differences of the map being computed (x → z for a density, z → x for a sample); blank cells are exactly zero.
code/ch18_normalizing_flows.py (made_init, maf_density, maf_sample, iaf_sample, iaf_density).