Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 17

What does a Gaussian prior on the weights say about functions?

Draw the weights and biases of f(x) = W2 φ(W1 x + b1) + b2 from N(0, α1²), N(0, β1²), N(0, α2²), N(0, β2²) and plot the resulting random functions. α1 makes the steps steeper, β1 moves their centres away from the origin, α2 scales the vertical range; with ReLU units and no biases the scales only rescale the output. The dashed band is plus and minus two exact prior standard deviations.

Activation

First-layer weight scale α1

First-layer bias scale β1

Output weight scale α2

Output bias scale β2

Hidden units H

Random draw

Try these

Ten functions are drawn with the demo kit's seeded generator; the table uses 400 further draws. The exact variance is H α2² E[φ(u)²] + β2² with u ~ N(0, α1²x² + β1²). The fraction of units whose step centre −b/w lies in [−1, 1] is (2/π) arctan(α1/β1).

In the lecture notes