Every method in this chapter is an answer to the same question: if the environment is gone, what stands in for it? The policy-gradient foundations chapter surveys the methods that still have an environment to query, and the trust-region chapter narrows to how a single update is bounded. This chapter takes the methods that give the environment up. Two substitutes are on offer and the split organises the whole literature: a fixed dataset, which the offline and preference-based methods learn from without ever collecting another transition, and a learned model, which the model-based methods roll forward in place of the simulator. The two look unrelated on the page and fail for one reason, so the table below is sorted by substitute and read for the column that matters, which is the last one.
| Method | Year | Stands in for the environment | Constraint on the policy | Critic | Where the region ends |
|---|---|---|---|---|---|
| CQL (Kumar et al. 2020) | 2020 | fixed dataset | conservative penalty pushing down off the data support | yes | the Bellman target keeps querying actions nobody collected, so on narrow or expert-only data the penalty suppresses good actions along with unseen ones |
| IQL (Kostrikov et al. 2022) | 2022 | fixed dataset | expectile regression on in-sample actions only | yes | never evaluates an unseen action, so it cannot exceed the best behaviour the dataset already contains |
| Decision Transformer (Chen et al. 2021) | 2021 | fixed dataset | none; return-conditioned sequence prediction | no | no Bellman backup means no stitching, so two mediocre trajectories cannot be recombined into a better one |
| Trajectory Transformer (Janner et al. 2021) | 2021 | fixed dataset | beam search over predicted trajectories | no | search quality is bounded by the sequence model, and compounding error grows with the planning horizon |
| RLHF with PPO (Ouyang et al. 2022) | 2022 | preference dataset via a learned reward model | KL penalty against the reference policy | yes | the reward model is itself fit on finite preferences and is over-optimised where they run out |
| DPO (Rafailov et al. 2023) | 2023 | preference dataset directly | implicit reward as a log-ratio to the reference | no | drifts in KL toward regions the preference set never covered, with no reward model to price the drift |
| World Models (Ha and Schmidhuber 2018) | 2018 | learned model (VAE plus MDN-RNN) | controller trained inside the rollout | no | the VAE bottleneck is fit for reconstruction, so it discards detail that is small in pixels and large in reward |
| MuZero (Schrittwieser et al. 2020) | 2020 | learned latent model | MCTS over the learned dynamics | yes | the latent predicts reward, value and policy accurately while corresponding to nothing inside the simulator |
| DreamerV3 (Hafner et al. 2023) | 2023 | learned latent model | actor-critic trained on imagined rollouts | yes | imagination compounds model error with horizon, which is why the rollout length is a tuned quantity rather than a free one |
| Episodic Planning Networks (Ritter et al. 2021) | 2021 | episodic memory of the current task | attention over stored transitions | no | memory is scoped to one task, so nothing transfers once the episode boundary is crossed |
Read down the last column and the two halves of the chapter stop looking like two literatures. CQL and World Models are solving the same problem in different notation. Both accept a substitute that is accurate on the region their data covers, both degrade silently rather than loudly outside it, and neither has a mechanism that reports when the boundary has been crossed. This is why the offline methods are characterised by support and the model-based methods by horizon: support is where a dataset stops being informative, horizon is where a model stops being accurate, and they are the same failure measured along different axes.
The column that separates the methods structurally is the critic. The methods without one are the ones that gave up on improvement over their data. Decision Transformer, Trajectory Transformer and DPO all replace value estimation with conditional sequence prediction, which buys stability and the entire transformer toolchain, and costs the ability to do better than what was shown. The methods with a critic keep that ability and pay for it with the off-support query that CQL's penalty and IQL's expectile are both built to survive. The chapter's comparison table revisits this split with the implementation detail attached, and the closing Reward-Is-Enough section asks whether any of it is more than instrumental.