Backpropagation through time multiplies the one-step Jacobians \(\mathbf J_j=\partial\mathbf s_j/\partial\mathbf s_{j-1}\) (Equation 15.22). The plot shows the spectral norm \(\|\partial\mathbf s_T/\partial\mathbf s_{T-k}\|_2\) against the lag \(k\) for four cells with 8 units driven by the same random input sequence of length 60. A linear RNN follows \(\rho^k\) exactly: it vanishes below \(\rho=1\) and explodes above it. A tanh RNN saturates and vanishes at any \(\rho\). A GRU and an LSTM have an additive path through the state whose strength is set by a gate: a large update-gate or forget-gate bias keeps the gradient alive over all 60 steps. Change the spectral radius, the weight scale and the gate bias and compare.
States: \(\mathbf h\) for the RNN and GRU, \((\mathbf h,\mathbf c)\) for the LSTM. The one-step Jacobian is evaluated in closed form along the trajectory driven by the input sequence (the formulas of the notes, gaps 15.8 and 15.9), and the product is accumulated from the last step backwards; the vertical axis is \(\log_{10}\) of its spectral norm. The dotted line is \(\rho^k\). The decay rate in the table is the slope of the curve between lags 10 and 59, in decades per step.
code/ch15_neural_networks_sequences.py (rnn_step, gru_step, lstm_step, state_jacobian, grad_flow, gf_curve).