With ReLU hidden units, each unit is either off (output 0) or on (output equal to its input), so once the on/off pattern of all units is fixed the network is an affine function of its input. The input space is therefore cut into linear regions, one per activation pattern that occurs. Left: the regions of a network on the square [−1, 1]2, one colour per pattern. A single layer of M units can make at most 1 + M + M(M−1)/2 regions in the plane, but stacking layers lets each layer fold the previous one's regions, so the maximum grows exponentially with depth (Montúfar et al. 2014). The sawtooth mode shows the cleanest case: L layers of 2 ReLUs give 2L linear pieces, which a one-layer network needs 2L − 1 units to match.
Random mode: weights N(0, 2/fan-in) (He initialisation) and biases N(0, s2), drawn from a seeded generator, layer by layer and unit by unit. Regions are counted as the distinct activation patterns at the centres of a 200 × 200 pixel grid, so very thin regions can be missed; for one layer the page also gives the exact count 1 + (lines that cross the square) + (crossings inside it).
code/ch06_deep_neural_networks.py (DeepReLU, count_regions, arrangement_regions, sawtooth). The region counts of the presets are reproduced exactly by count_regions.