Each node starts with one number. A message-passing layer replaces it by (1 − a) times its own value plus a times the average of its neighbours' values, the simplest version of the update h′i = φ(U hi + ∑j V hj) without the learned weights. After K layers a node has heard only from nodes at most K steps away (the gold ring), and the features drift towards a common value: over-smoothing. A global node, or the complete graph that a transformer uses, shortens the distances at once. Click a node to choose it.
Features start at hi = cos(2.4 i). The Dirichlet energy is the sum over edges of (hi − hj)2; the plot shows its ratio to the starting value. The receptive field of node i after K layers is the set of nodes j with a walk of length at most K to i (the support of (I + A)K).
code/ch16_deep_neural_networks.py (graph_adj, propagation_matrix, dirichlet_energy); see the code index.