A corridor has six states; the agent starts at the left end and chooses left or right with a tabular softmax policy; the right end pays 1 per step, and a constant offset is added to every reward, which changes nothing about the best policy but makes all returns large. The REINFORCE gradient estimate \(\sum_t\gamma^t(G_t-b(s_t))\nabla\log\pi(a_t\mid s_t)\) is unbiased for every baseline \(b\), but its spread is not: with \(b=0\) it is dominated by noise. Three runs of the same algorithm differ only in the baseline: none, a constant (a running average of the return from the start), and a state baseline \(V_w(s)\) learned by Monte Carlo regression (Algorithm 35.2). The upper plot shows the exact expected return of the policy during learning; the lower one the standard deviation of a one-episode gradient estimate at the starting policy.
Corridor of 6 states, \(\gamma=0.9\), episodes of 80 steps (so the discounted tail is negligible), softmax policy with one parameter per state and action, initial parameters 0. The return is plotted exactly (the policy is evaluated by solving the Bellman equations), not estimated. The state baseline is regressed on the returns from the first 20 steps of each episode with step 0.1; the constant baseline is an exponential moving average (0.9) of the return from the start. The bars use 6000 simulated episodes at \(\theta=0\); for the state baseline they use the exact value function \(V_\pi\).
code/ch35_reinforcement_learning.py (reinforce_batch, reinforce_learning_curve).