Trajectories come from a behaviour policy (uniform over two actions) on a small random MDP with 3 states; we want the value of a target policy that takes action 0 with probability \(p_e\). The importance weight of a trajectory is the product of the per-step ratios \(\rho_t=\pi_e/\pi_b\); its variance is \(m^T-1\) with \(m=\mathbb E_b[\rho^2]\), exponential in the horizon \(T\). Four estimators are compared by their root-mean-square error against \(T\): IS (35.47), per-decision IS (35.49), self-normalised IS (WIS) and the doubly robust estimator (35.53) with a reward model whose \(Q\) values are too high by a constant. The right panel is the variance of the trajectory weight.
MDP: 3 states, 2 actions, random transition probabilities and mean rewards in [0,1], reward noise s.d. 0.1, \(\gamma=0.99\), start in state 0; it is the one used for Figure 35.14. The target value is the exact value of \(\pi_e\) for the horizon \(T\). The reward model is \(\hat Q=Q_{\pi_e}+b\,(1-\gamma^k)/(3(1-\gamma))\) with \(k\) steps to go. RMSE is over the repetitions; the IS curve is very heavy-tailed, so it is noisy at long horizons.
code/ch35_reinforcement_learning.py (ope_env, ope_estimates, dr_recursive); the MDP is in data/ch35-ope-env.json.