Code that evaluates a function is a graph of elementary operations vi (8.50)–(8.56). Forward mode carries a tangent v̇i = ∂vi/∂xj along with each value, summing over the parents (8.57): one pass gives one column of the Jacobian. Reverse mode first records the values, then sends adjoints v̄i = ∂f/∂vi backwards, summing over the children (8.69): one pass gives one row, that is the whole gradient of one output. The numbers on the edges are the local derivatives ∂vi/∂vj.
Highlighted node: the one just evaluated. In reverse mode the first half of the steps is the forward sweep that stores every vi (the memory cost of reverse mode); the second half is the backward sweep. The soft ReLU is ζ(a) = ln(1 + ea) (8.46), whose derivative is the logistic sigmoid.
code/ch08_backpropagation.py (graph_eval, the classes Dual and Var, swell_table); the values here equal graph_eval, number for number.