The data have a mean sin 3x but a noise level that grows towards the ends of the interval. The network maps x through one tanh hidden layer to two outputs: a mean μ(x) (linear head) and a standard deviation σ(x) = softplus(.) + 0.001 (softplus head), trained by full-batch Adam on the Gaussian negative log-likelihood. Switch the second head off to force one constant σ and compare the fits and the loss.
Loss: mean over the 150 points of ½ log(2πσ2) + (y − μ)2/(2σ2), as for (16.22). The initial weights are fixed, so every run is reproducible. With the constant-variance head the second head's weights are frozen at zero and σ is a single learned number.
code/ch16_deep_neural_networks.py (HeteroMLP, hetero_data); see the code index.