The function f(x1, x2) = x2 ex1 √(x1 + x2 ex1) of Equation (13.56) is evaluated by five elementary steps (13.57)–(13.61). Reverse mode visits the nodes in reverse topological order and attaches to each node j the adjoint ∂f/∂xj = Σk∈children (∂f/∂xk)(∂xk/∂xj) (13.67): one backward pass gives both partial derivatives. Forward mode pushes one tangent ∂xj/∂xi through the graph and needs one pass per input. Below the graph, the cost of the full Jacobian of a chain of layers shows why the number of inputs n and outputs m decides which mode to use.
Every node is an elementary operation with a known local derivative. The adjoint of a node that feeds two children, such as x4, is the sum of two path contributions (13.62). The chain has three n×n layers and a last m×n layer; a matrix-vector product with an a×b matrix costs ab multiply-adds. Forward mode runs n Jacobian-vector products (each a forward pass plus a tangent pass), reverse mode one forward pass and m vector-Jacobian products.
code/ch13_neural_networks_tabular.py (graph_value_and_adjoints, chain_jacobian_forward, chain_jacobian_reverse).