Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 16 (supplement)

How far does a message travel?

Each node starts with one number. A message-passing layer replaces it by (1 − a) times its own value plus a times the average of its neighbours' values, the simplest version of the update h′i = φ(U hi + ∑j V hj) without the learned weights. After K layers a node has heard only from nodes at most K steps away (the gold ring), and the features drift towards a common value: over-smoothing. A global node, or the complete graph that a transformer uses, shortens the distances at once. Click a node to choose it.

Graph

Layers K

Mixing a (weight on the neighbour average)

Selected node

Try these

Features start at hi = cos(2.4 i). The Dirichlet energy is the sum over edges of (hi − hj)2; the plot shows its ratio to the starting value. The receptive field of node i after K layers is the set of nodes j with a walk of length at most K to i (the support of (I + A)K).

In the lecture notes