Actor-Critic and Distributed RL: DDPG, TD3, SAC, and Beyond

This part-level leaderboard zooms into the actor-critic and distributed RL families specifically; the broader cross-part RL leaderboard lives in rl/01 Related Work and is not duplicated here. Columns track design axes that distinguish methods within these two families rather than across the whole book: policy class (deterministic, stochastic Gaussian, max-entropy, V-trace-corrected on-policy), the number of critic networks (a load-bearing hyperparameter for overestimation-bias control[fn:s0-overestimation]), and the two canonical scoreboards (MuJoCo HalfCheetah-v2 reward at 1M env steps for continuous control; Atari-57 median human-normalized score for distributed agents). Entries marked n.r. are not reported in the original paper because the method targets only the other domain; entries marked $\dagger$ come from third-party reproductions[fn:s0-reproductions] rather than the original paper.

Actor-critic and distributed-RL leaderboard. Each row's citation names the method's source paper, not necessarily the source of that row's benchmark numbers. Type abbreviations are DPG for deterministic policy gradient, Stoch-G for stochastic Gaussian policy, MaxEnt for maximum-entropy soft actor-critic, A-C async for asynchronous advantage actor-critic, V-trace for truncated importance-sampling correction (Espeholt 2018), Ape-X for centralized prioritized replay (Horgan 2018), and NGU for the Never-Give-Up intrinsic-reward population. In the off-policy column, *Yes* uses replay, *partial* uses a small replay or a near-on-policy correction, and No is fully on-policy. Critic count tracks the overestimation-control axis, with 1 critic (DDPG, A2C/A3C, ACER, IMPALA) the high-bias regime and 2 critics with a min-target (TD3, SAC, MPO) the variance-bias frontier. HalfCheetah-v2 reward is at 1M env steps, and the Atari column is the median human-normalized score from the original paper unless noted. Two caveats on the Atari column. The A3C entry of 344% is the *mean* human-normalized score of the one-day run in Table 1 of (Mnih et al. 2016), not a median (its median is 68.2% at one day and 82.2% at four days), so that single cell is not on the same footing as IMPALA's 191.8% and R2D2's 1920%, which genuinely are medians. The A2C entry is a third-party reproduction rather than a reported result. Dashes mark unreported settings.
Method	Year	Type	Off-policy	Critics	HalfCheetah-v2 (1M)	Atari-57 median HNS
DDPG (Lillicrap et al. 2019)	2015	DPG	Yes	1	$\sim$ 3300	n.r.
NAF (Gu et al. 2016)	2016	DPG (quad. Q)	Yes	1 (Q=V+A)	$\sim$ 2800	n.r.
A3C (Mnih et al. 2016)	2016	A-C async	No	1 (V)	n.r.	344%
A2C (sync A3C) (Mnih et al. 2016)	2017	A-C sync	No	1 (V)	n.r.	$\sim$ 300% $\dagger$
ACER (Wang et al. 2017)	2017	Stoch-G + Retrace	partial	1 (Q)	n.r.	$\sim$ A3C-median
TD3 (Fujimoto et al. 2018)	2018	DPG, twin-min	Yes	2 (clipped)	9636	n.r.
SAC v2 (Haarnoja et al. 2019)	2018	MaxEnt Stoch-G	Yes	2 (min)	$\sim$ 11000	n.r.
MPO (Abdolmaleki et al. 2018)	2018	EM Stoch-G	Yes	2 (target+online)	$\sim$ 9000	n.r.
IMPALA (Espeholt et al. 2018)	2018	A-C + V-trace	partial	1 (V)	n.r.	191.8%
R2D2 (Kapturowski et al. 2018)	2019	Q-LSTM + Ape-X	Yes	1 (Q dueling)	n.r.	1920%
Agent57 (Badia et al. 2020)	2020	NGU + R2D2	Yes	2 ( $Q_e, Q_i$ )	n.r.	superhuman 57/57

The continuous-control column tracks a clean three-step monotone improvement driven by overestimation-bias control[fn:s0-overestimation]. DDPG's single Q-network and OU exploration plateau near 3300; TD3 nearly triples this to 9636 by combining clipped twin critics, delayed actor updates, and target-policy smoothing; SAC pushes further to roughly 11000 by replacing the deterministic actor with a max-entropy stochastic policy whose temperature $\alpha$ is auto-tuned to match a target entropy. MPO arrives at a similar ballpark ( $\sim$ 9000) via a different route: EM-style decoupled policy improvement with a non-parametric weighting step, which is empirically less sensitive to the learning rate but rarely beats SAC on raw HalfCheetah numbers. The critic-count axis is the single most predictive design choice on this benchmark: every method that crosses the 9000 line uses two critics with a min-target, and every method that stalls below 5000 uses one.

The distributed column is a different scoreboard with a different bottleneck. A3C ran 16 CPU workers asynchronously to reach 344% mean human-normalized score on Atari after one day (68% median), and the A2C synchronous variant matched it on a single GPU once vectorized environments became cheap. ACER folded experience replay into A3C with truncated importance sampling and Retrace, but its Atari median was only marginally above A3C and it lost the dominant-on-policy-AC title to PPO/SAC within six months. IMPALA's V-trace correction broke the on-policy throughput ceiling: by sending behaviour-policy probabilities alongside actions and correcting at the learner, IMPALA scaled linearly to roughly 500 actors at 250k FPS, but its median 191.8% on Atari-57 trailed the prioritized-replay Ape-X / R2D2 line because off-policy correction is too aggressive at moderate staleness. R2D2 closed that gap to 1920% by adding LSTM Q-networks with the burn-in trick over Ape-X-style replay, accepting mild staleness as a regularizer rather than correcting for it. Agent57 then layered NGU intrinsic rewards plus a 32-arm meta-controller on top of R2D2, becoming the first agent superhuman on all 57 games at the cost of $\sim$ USD 10k compute per game. The actor-critic and distributed lines converge in Agent57: it is simultaneously a two-critic actor-critic (separate $Q_e, Q_i$ heads) and a fully distributed agent (512 actors, single TPU learner), making it the natural terminal entry of this part.

References

Abdolmaleki, Abbas, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. 2018. Maximum a Posteriori Policy Optimisation. arXiv. https://doi.org/10.48550/arXiv.1806.06920.

Badia, Adrià Puigdomènech, Bilal Piot, Steven Kapturowski, et al. 2020. Agent57: Outperforming the Atari Human Benchmark. arXiv. https://doi.org/10.48550/arXiv.2003.13350.

Espeholt, Lasse, Hubert Soyer, Remi Munos, et al. 2018. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. arXiv. https://doi.org/10.48550/arXiv.1802.01561.

Fujimoto, Scott, Herke van Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Critic Methods. arXiv. https://doi.org/10.48550/arXiv.1802.09477.

Gu, Shixiang, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. 2016. Continuous Deep Q-Learning with Model-Based Acceleration. arXiv. https://doi.org/10.48550/arXiv.1603.00748.

Haarnoja, Tuomas, Aurick Zhou, Kristian Hartikainen, et al. 2019. Soft Actor-Critic Algorithms and Applications. arXiv. https://doi.org/10.48550/arXiv.1812.05905.

Kapturowski, Steven, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. 2018. "Recurrent Experience Replay in Distributed Reinforcement Learning." September. https://openreview.net/forum?id=r1lyTjAqYX.

Lillicrap, Timothy P., Jonathan J. Hunt, Alexander Pritzel, et al. 2019. Continuous Control with Deep Reinforcement Learning. arXiv. https://doi.org/10.48550/arXiv.1509.02971.

Mnih, Volodymyr, Adrià Puigdomènech Badia, Mehdi Mirza, et al. 2016. Asynchronous Methods for Deep Reinforcement Learning. arXiv. https://doi.org/10.48550/arXiv.1602.01783.

Wang, Ziyu, Victor Bapst, Nicolas Heess, et al. 2017. Sample Efficient Actor-Critic with Experience Replay. arXiv. https://doi.org/10.48550/arXiv.1611.01224.