Trust-Region Policy Methods: TRPO, PPO, and GRPO

The trust-region family is the longest-running active line in policy optimization, persisting from Schulman's 2015 KL-constrained natural gradient through the clipped surrogate that defines current RLHF pipelines. Where the broader RL leaderboard in the policy-gradient foundations chapter surveys foundations, off-policy actor-critics, distributed and model-based methods together, this chapter narrows to one design axis. Each variant enforces or approximates a trust region on the policy update in a different way. TRPO enforces $D_{KL}\!\le\!\delta$ exactly via line-search. PPO-KL adds $-\beta D_{KL}$ as a soft penalty with $\beta$ adapted by target-KL feedback. PPO-clip drops the KL apparatus and clips the ratio $r_t \in [1-\epsilon, 1+\epsilon]$ , an empirically softer trust region whose measured KL routinely exceeds the bound $\epsilon$ implies. ACKTR substitutes K-FAC for conjugate gradient. GRPO removes the critic entirely. The methods below differ on four dimensions covered in detail across the chapter: (1) how the KL bound is imposed (hard line-search, soft penalty, clipped ratio, or none), (2) the surrogate objective form, (3) whether a learned critic is required, and (4) the domain (MuJoCo locomotion, Atari, MOBA games, or LLM reasoning) where the variant was first deployed.

Trust-region methods covered in this chapter, arranged by how the trust region is enforced. KL bound records the enforcement mechanism, where $\delta$ is the hard radius for line-search variants, $\beta$ the soft-penalty coefficient and $\epsilon$ the clip half-width. Surrogate type contrasts the importance-weighted advantage form ( $r_t \hat{A}$ ), its clipped composition, and the dual-clip three-way max. Critic indicates whether a learned value head is required. **The last column is not a leaderboard and its HalfCheetah figures are illustrative placeholders rather than measured results.** The 2015 TRPO paper reports Swimmer, Hopper and Walker plus Atari and has no HalfCheetah task, the GAE paper reports cart-pole, 3D biped, quadruped and 3D humanoid, and no common HalfCheetah-at-1M protocol is shared across the four MuJoCo rows, so the five values form a composite that no single source supports. Only the DPPO, dual-clip and GRPO entries reproduce quantities their own papers state. The per-row citation names the method's source paper, never the row's numbers. For an actual head-to-head between PPO and TRPO under matched code-level settings, see (Engstrom et al. 2020), discussed in the implementation-details section.
Method	Year	KL bound	Surrogate type	Critic	Domain	Reported figure (illustrative)
TRPO (Schulman et al. 2015)	2015	hard, $\delta$ =0.01 (CG+line-search)	$r_t \hat{A}$ (constrained max)	yes (V)	MuJoCo, Atari	HalfCheetah ~4500 @ 1M steps
GAE+TRPO (Schulman et al. 2018)	2016	hard, $\delta$ =0.01	$r_t \hat{A}^{GAE(\lambda=0.96)}$	yes (V, TR-clipped)	MuJoCo	HalfCheetah ~4800 @ 1M steps
ACKTR (Wu et al. 2017)	2017	soft, $\delta$ =10 $^{-3}$ (K-FAC)	$r_t \hat{A}$ , K-FAC $F^{-1}$	yes (V)	MuJoCo, Atari	HalfCheetah ~4900 @ 1M steps
PPO-clip (Schulman et al. 2017)	2017	implicit, $\epsilon$ =0.2 (clip)	$\min(r_t\hat A, \mathrm{clip}(r_t)\hat A)$	yes (V)	MuJoCo, Atari	HalfCheetah ~5800 @ 1M steps
PPO-KL (Schulman et al. 2017)	2017	soft, $\beta$ adaptive (target=0.01)	$r_t\hat A - \beta D_{KL}$	yes (V)	MuJoCo	HalfCheetah ~3800 @ 1M steps
DPPO (Heess et al. 2017)	2017	soft, $\beta$ asym (1.5/0.67)	$r_t\hat A - \beta D_{KL}$ , sync-grad	yes (V)	MuJoCo terrain	humanoid 80% asymp. @ 1B steps, 5d/128 workers
Dual-clip PPO (Ye et al. 2020)	2020	implicit, $\epsilon$ =0.2, $c$ =3	$\max(\min(r_t\hat A,\mathrm{clip}\hat A), c\hat A)$ when $\hat A<0$	yes (V)	Honor of Kings (MOBA)	superhuman 1v1; divergence 15% $\to$ 3%
GRPO (Shao et al. 2024)	2024	soft, $\beta$ =0.04 (unbiased est.)	clip-PPO + group-relative $\hat A$	none (group baseline)	LLM math reasoning	MATH 51.7% / GSM8K 88.2% (DeepSeekMath 7B)

Reading the table chronologically traces the trust-region trade-off curve, with the caveat that the last column is a composite and carries no ranking. TRPO 2015 sets the operational ceiling on KL enforcement with a hard $\delta = 0.01$ bound enforced by conjugate gradient on the Fisher-vector product plus a backtracking line search, giving the monotonic-improvement guarantee at the cost of second-order machinery and ~10 CG iterations per update. ACKTR keeps the natural-gradient direction but replaces CG with K-FAC's block-diagonal Kronecker factorization (Wu et al. 2017). Its reported comparison is against A2C and TRPO on MuJoCo and Atari, on sample efficiency per environment step and on added per-update cost over first-order SGD. It predates PPO by about a month and reports no wall-clock comparison against it, so any PPO-parity claim for ACKTR is a later addition rather than a result in the paper. PPO-clip 2017 then drops the KL apparatus entirely in favor of a clipped ratio surrogate, and the empirical reality (already noted in (Engstrom et al. 2020)) is that measured KL routinely exceeds what $\epsilon$ would imply. What actually limits divergence is the small epoch count plus learning-rate annealing, with the clip acting as a soft attenuator on outlier gradients. PPO-KL is the original paper's adaptive-penalty variant. It underperforms the clip on standard MuJoCo and is rarely used outside the DPPO terrain-locomotion regime where the asymmetric $\beta$ feedback was load-bearing.

The 2020+ rows extend the family to two new domains where vanilla clipping breaks. Dual-clip PPO addresses an off-policy-PPO failure mode at large importance ratios by adding a lower bound $c\hat{A}_t$ when $\hat{A}_t < 0$ , reducing divergence from roughly 15% to 3% of seeds in the JueWu Honor of Kings stack without changing final policy quality. GRPO eliminates the critic entirely by sampling $G\!=\!64$ outputs per prompt and using within-group reward normalization as the advantage estimate; this halves training memory for LLMs (where critic-of-policy-size is prohibitive) and uses an unbiased KL estimator $\pi_{ref}/\pi_\theta - \log(\pi_{ref}/\pi_\theta) - 1$ rather than the log-ratio form, becoming the de facto RLHF backbone for reasoning models. The decade-long arc converges on a single recipe, namely clipped or KL-penalized importance-weighted surrogates with code-level optimizations (Engstrom et al. 2020) doing more empirical work than the algorithmic differences, and the chapter on implementation details covers which optimizations dominate.

References

Engstrom, Logan, Andrew Ilyas, Shibani Santurkar, et al. 2020. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. arXiv. https://doi.org/10.48550/arXiv.2005.12729.

Heess, Nicolas, Dhruva TB, Srinivasan Sriram, et al. 2017. Emergence of Locomotion Behaviours in Rich Environments. arXiv. https://doi.org/10.48550/arXiv.1707.02286.

Schulman, John, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2015. Trust Region Policy Optimization. arXiv. https://doi.org/10.48550/arXiv.1502.05477.

Schulman, John, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2018. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv. https://doi.org/10.48550/arXiv.1506.02438.

Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv. https://doi.org/10.48550/arXiv.1707.06347.

Shao, Zhihong, Peiyi Wang, Qihao Zhu, et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv. https://doi.org/10.48550/arXiv.2402.03300.

Wu, Yuhuai, Elman Mansimov, Shun Liao, Roger Grosse, and Jimmy Ba. 2017. Scalable Trust-Region Method for Deep Reinforcement Learning Using Kronecker-Factored Approximation. arXiv. https://doi.org/10.48550/arXiv.1708.05144.

Ye, Deheng, Zhao Liu, Mingfei Sun, et al. 2020. "Mastering Complex Control in MOBA Games with Deep Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence 34: 6672-79. http://arxiv.org/abs/1912.09729.