Hado van Hasselt, Arthur Guez, David Silver
The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Atari Games | Atari 2600 Boxing | Score | 88 | DQN noop |
| Atari Games | Atari 2600 Boxing | Score | 79.2 | Prior+Duel hs |
| Atari Games | Atari 2600 Boxing | Score | 73.5 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Boxing | Score | 70.3 | DQN hs |
| Atari Games | Atari 2600 Double Dunk | Score | -6 | DQN hs |
| Atari Games | Atari 2600 Double Dunk | Score | -6.6 | DQN noop |
| Atari Games | Atari 2600 Double Dunk | Score | -0.3 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Double Dunk | Score | -10.7 | Prior+Duel hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 3085.6 | DQN noop |
| Atari Games | Atari 2600 Ms. Pacman | Score | 1241.3 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 1092.3 | DQN hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 1007.8 | Prior+Duel hs |
| Atari Games | Atari 2600 Centipede | Score | 5570.2 | Prior+Duel hs |
| Atari Games | Atari 2600 Centipede | Score | 4657.7 | DQN noop |
| Atari Games | Atari 2600 Centipede | Score | 3973.9 | DQN hs |
| Atari Games | Atari 2600 Centipede | Score | 3853.5 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Tutankham | Score | 108.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Tutankham | Score | 92.2 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Tutankham | Score | 68.1 | DQN noop |
| Atari Games | Atari 2600 Tutankham | Score | 45.6 | DQN hs |
| Atari Games | Atari 2600 Freeway | Score | 30.8 | DQN noop |
| Atari Games | Atari 2600 Freeway | Score | 28.8 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Freeway | Score | 28.2 | Prior+Duel hs |
| Atari Games | Atari 2600 Freeway | Score | 26.9 | DQN hs |
| Atari Games | Atari 2600 Pong | Score | 19.5 | DQN noop |
| Atari Games | Atari 2600 Pong | Score | 19.1 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Pong | Score | 18.4 | Prior+Duel hs |
| Atari Games | Atari 2600 Pong | Score | 18 | DQN hs |
| Atari Games | Atari 2600 Enduro | Score | 2223.9 | Prior+Duel hs |
| Atari Games | Atari 2600 Enduro | Score | 1216.6 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Enduro | Score | 729 | DQN noop |
| Atari Games | Atari 2600 Enduro | Score | 626.7 | DQN hs |
| Atari Games | Atari 2600 Krull | Score | 8422.3 | DQN noop |
| Atari Games | Atari 2600 Krull | Score | 7658.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Krull | Score | 6796.1 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Krull | Score | 6206 | DQN hs |
| Atari Games | Atari 2600 Breakout | Score | 385.5 | DQN noop |
| Atari Games | Atari 2600 Breakout | Score | 368.9 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Breakout | Score | 354.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Breakout | Score | 354.5 | DQN hs |
| Atari Games | Atari 2600 Frostbite | Score | 4038.4 | Prior+Duel hs |
| Atari Games | Atari 2600 Frostbite | Score | 1448.1 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Frostbite | Score | 797.4 | DQN noop |
| Atari Games | Atari 2600 Frostbite | Score | 496.1 | DQN hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 47 | DQN hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 42 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 24 | Prior+Duel hs |
| Atari Games | Atari 2600 Gopher | Score | 105148.4 | Prior+Duel hs |
| Atari Games | Atari 2600 Gopher | Score | 15253 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Gopher | Score | 8777.4 | DQN noop |
| Atari Games | Atari 2600 Gopher | Score | 8190.4 | DQN hs |
| Atari Games | Atari 2600 Space Invaders | Score | 8978 | Prior+Duel hs |
| Atari Games | Atari 2600 Space Invaders | Score | 2628.7 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Space Invaders | Score | 1692.3 | DQN noop |
| Atari Games | Atari 2600 Space Invaders | Score | 1293.8 | DQN hs |
| Atari Games | Atari 2600 James Bond | Score | 768.5 | DQN noop |
| Atari Games | Atari 2600 James Bond | Score | 697.5 | DQN hs |
| Atari Games | Atari 2600 James Bond | Score | 585 | Prior+Duel hs |
| Atari Games | Atari 2600 James Bond | Score | 573 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Amidar | Score | 978 | DQN noop |
| Atari Games | Atari 2600 Amidar | Score | 178.4 | DQN hs |
| Atari Games | Atari 2600 Amidar | Score | 169.1 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Tennis | Score | 12.2 | DQN noop |
| Atari Games | Atari 2600 Tennis | Score | 11.1 | DQN hs |
| Atari Games | Atari 2600 Tennis | Score | -7.8 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Tennis | Score | -13.2 | Prior+Duel hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 127853 | Prior+Duel hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 113782 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 110763 | DQN noop |
| Atari Games | Atari 2600 Crazy Climber | Score | 98128 | DQN hs |
| Atari Games | Atari 2600 Asteroids | Score | 1458.7 | DQN hs |
| Atari Games | Atari 2600 Asteroids | Score | 1364.5 | DQN noop |
| Atari Games | Atari 2600 Asteroids | Score | 1193.2 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Asteroids | Score | 1021.9 | Prior+Duel hs |
| Atari Games | Atari 2600 Gravitar | Score | 473 | DQN noop |
| Atari Games | Atari 2600 Gravitar | Score | 298 | DQN hs |
| Atari Games | Atari 2600 Gravitar | Score | 200.5 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Gravitar | Score | 167 | Prior+Duel hs |
| Atari Games | Atari 2600 Time Pilot | Score | 6608 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Time Pilot | Score | 4871 | Prior+Duel hs |
| Atari Games | Atari 2600 Time Pilot | Score | 4870 | DQN noop |
| Atari Games | Atari 2600 Time Pilot | Score | 4786 | DQN hs |
| Atari Games | Atari 2600 Demon Attack | Score | 73371.3 | Prior+Duel hs |
| Atari Games | Atari 2600 Demon Attack | Score | 69803.4 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Demon Attack | Score | 12550.7 | DQN hs |
| Atari Games | Atari 2600 Demon Attack | Score | 12149.4 | DQN noop |
| Atari Games | Atari 2600 Battle Zone | Score | 30650 | Prior+Duel hs |
| Atari Games | Atari 2600 Battle Zone | Score | 29900 | DQN noop |
| Atari Games | Atari 2600 Battle Zone | Score | 24740 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Battle Zone | Score | 23750 | DQN hs |
| Atari Games | Atari 2600 Beam Rider | Score | 37412.2 | Prior+Duel hs |
| Atari Games | Atari 2600 Beam Rider | Score | 17417.2 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Beam Rider | Score | 9743.2 | DQN hs |
| Atari Games | Atari 2600 Beam Rider | Score | 8627.5 | DQN noop |
| Atari Games | Atari 2600 Asterix | Score | 16837 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Asterix | Score | 4359 | DQN noop |
| Atari Games | Atari 2600 Asterix | Score | 3170.5 | DQN hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 37484 | Prior+Duel hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 30207 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 26059 | DQN noop |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 20882 | DQN hs |
| Atari Games | Atari 2600 Bowling | Score | 69.6 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Bowling | Score | 56.5 | DQN hs |
| Atari Games | Atari 2600 Bowling | Score | 50.4 | DQN noop |
| Atari Games | Atari 2600 Bowling | Score | 50.4 | Prior+Duel hs |
| Atari Games | Atari 2600 Kangaroo | Score | 11204 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Kangaroo | Score | 7259 | DQN noop |
| Atari Games | Atari 2600 Kangaroo | Score | 4496 | DQN hs |
| Atari Games | Atari 2600 Kangaroo | Score | 861 | Prior+Duel hs |
| Atari Games | Atari 2600 Assault | Score | 6060.8 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Assault | Score | 4280.4 | DQN noop |
| Atari Games | Atari 2600 Assault | Score | 3489.3 | DQN hs |
| Atari Games | Atari 2600 Alien | Score | 1620 | DQN noop |
| Atari Games | Atari 2600 Alien | Score | 1033.4 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Alien | Score | 634 | DQN hs |
| Atari Games | Atari 2600 Fishing Derby | Score | 17 | Prior+Duel hs |
| Atari Games | Atari 2600 Fishing Derby | Score | 3.2 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Fishing Derby | Score | -1.6 | DQN hs |
| Atari Games | Atari 2600 Fishing Derby | Score | -4.9 | DQN noop |
| Atari Games | Atari 2600 Seaquest | Score | 14498 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Seaquest | Score | 5860.6 | DQN noop |
| Atari Games | Atari 2600 Seaquest | Score | 4216.7 | DQN hs |
| Atari Games | Atari 2600 Seaquest | Score | 1431.2 | Prior+Duel hs |
| Atari Games | Atari 2600 Chopper Command | Score | 8058 | Prior+Duel hs |
| Atari Games | Atari 2600 Chopper Command | Score | 6126 | DQN noop |
| Atari Games | Atari 2600 Chopper Command | Score | 5017 | DQN hs |
| Atari Games | Atari 2600 Chopper Command | Score | 3495 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Video Pinball | Score | 447408.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Video Pinball | Score | 367823.7 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Video Pinball | Score | 196760.4 | DQN noop |
| Atari Games | Atari 2600 Video Pinball | Score | 154414.1 | DQN hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 10471 | Prior+Duel hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 6201 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 2704 | DQN noop |
| Atari Games | Atari 2600 Wizard of Wor | Score | 1609 | DQN hs |
| Atari Games | Atari 2600 Zaxxon | Score | 11320 | Prior+Duel hs |
| Atari Games | Atari 2600 Zaxxon | Score | 8593 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Zaxxon | Score | 5363 | DQN noop |
| Atari Games | Atari 2600 Zaxxon | Score | 4412 | DQN hs |
| Atari Games | Atari 2600 Robotank | Score | 63.9 | DQN noop |
| Atari Games | Atari 2600 Robotank | Score | 59.1 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Robotank | Score | 58.7 | DQN hs |
| Atari Games | Atari 2600 Robotank | Score | 24.7 | Prior+Duel hs |
| Atari Games | Atari 2600 Name This Game | Score | 13637.9 | Prior+Duel hs |
| Atari Games | Atari 2600 Name This Game | Score | 8960.3 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Name This Game | Score | 8207.8 | DQN noop |
| Atari Games | Atari 2600 Name This Game | Score | 6738.8 | DQN hs |
| Atari Games | Atari 2600 Star Gunner | Score | 127073 | Prior+Duel hs |
| Atari Games | Atari 2600 Star Gunner | Score | 58365 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Star Gunner | Score | 54282 | DQN noop |
| Atari Games | Atari 2600 Star Gunner | Score | 52970 | DQN hs |
| Atari Games | Atari 2600 Ice Hockey | Score | 0.5 | Prior+Duel hs |
| Atari Games | Atari 2600 Ice Hockey | Score | -1.6 | DQN hs |
| Atari Games | Atari 2600 Ice Hockey | Score | -1.9 | DQN noop |
| Atari Games | Atari 2600 Ice Hockey | Score | -2.5 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Berzerk | Score | 2178.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Berzerk | Score | 1011.1 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Berzerk | Score | 585.6 | DQN noop |
| Atari Games | Atari 2600 Berzerk | Score | 493.4 | DQN hs |
| Atari Games | Atari 2600 Atlantis | Score | 423252 | Prior+Duel hs |
| Atari Games | Atari 2600 Atlantis | Score | 319688 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Atlantis | Score | 292491 | DQN hs |
| Atari Games | Atari 2600 Atlantis | Score | 279987 | DQN noop |
| Atari Games | Atari 2600 HERO | Score | 20437.8 | DQN noop |
| Atari Games | Atari 2600 HERO | Score | 15459.2 | Prior+Duel hs |
| Atari Games | Atari 2600 HERO | Score | 14992.9 | DQN hs |
| Atari Games | Atari 2600 HERO | Score | 14892.5 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Bank Heist | Score | 1004.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Bank Heist | Score | 886 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Bank Heist | Score | 455 | DQN noop |
| Atari Games | Atari 2600 Bank Heist | Score | 312.7 | DQN hs |
| Atari Games | Atari 2600 Venture | Score | 163 | DQN noop |
| Atari Games | Atari 2600 Venture | Score | 136 | DQN hs |
| Atari Games | Atari 2600 Venture | Score | 29 | Prior+Duel hs |
| Atari Games | Atari 2600 Venture | Score | 21 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Private Eye | Score | 1277.6 | Prior+Duel hs |
| Atari Games | Atari 2600 Private Eye | Score | 207.9 | DQN hs |
| Atari Games | Atari 2600 Private Eye | Score | 146.7 | DQN noop |
| Atari Games | Atari 2600 Private Eye | Score | -575.5 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Q*Bert | Score | 14063 | Prior+Duel hs |
| Atari Games | Atari 2600 Q*Bert | Score | 13117.3 | DQN noop |
| Atari Games | Atari 2600 Q*Bert | Score | 11020.8 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Q*Bert | Score | 9271.5 | DQN hs |
| Atari Games | Atari 2600 River Raid | Score | 16496.8 | Prior+Duel hs |
| Atari Games | Atari 2600 River Raid | Score | 10838.4 | DDQN (tuned) hs |
| Atari Games | Atari 2600 River Raid | Score | 7377.6 | DQN noop |
| Atari Games | Atari 2600 River Raid | Score | 4748.5 | DQN hs |
| Atari Games | Atari 2600 Road Runner | Score | 54630 | Prior+Duel hs |
| Atari Games | Atari 2600 Road Runner | Score | 43156 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Road Runner | Score | 39544 | DQN noop |
| Atari Games | Atari 2600 Road Runner | Score | 35215 | DQN hs |
| Atari Games | Atari 2600 Up and Down | Score | 22681.3 | Prior+Duel hs |
| Atari Games | Atari 2600 Up and Down | Score | 19086.9 | DDQN (tuned) hs |
| Atari Games | Atari 2600 Up and Down | Score | 9989.9 | DQN noop |
| Atari Games | Atari 2600 Up and Down | Score | 8038.5 | DQN hs |
| Video Games | Atari 2600 Boxing | Score | 88 | DQN noop |
| Video Games | Atari 2600 Boxing | Score | 79.2 | Prior+Duel hs |
| Video Games | Atari 2600 Boxing | Score | 73.5 | DDQN (tuned) hs |
| Video Games | Atari 2600 Boxing | Score | 70.3 | DQN hs |
| Video Games | Atari 2600 Double Dunk | Score | -6 | DQN hs |
| Video Games | Atari 2600 Double Dunk | Score | -6.6 | DQN noop |
| Video Games | Atari 2600 Double Dunk | Score | -0.3 | DDQN (tuned) hs |
| Video Games | Atari 2600 Double Dunk | Score | -10.7 | Prior+Duel hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 3085.6 | DQN noop |
| Video Games | Atari 2600 Ms. Pacman | Score | 1241.3 | DDQN (tuned) hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 1092.3 | DQN hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 1007.8 | Prior+Duel hs |
| Video Games | Atari 2600 Centipede | Score | 5570.2 | Prior+Duel hs |
| Video Games | Atari 2600 Centipede | Score | 4657.7 | DQN noop |
| Video Games | Atari 2600 Centipede | Score | 3973.9 | DQN hs |
| Video Games | Atari 2600 Centipede | Score | 3853.5 | DDQN (tuned) hs |
| Video Games | Atari 2600 Tutankham | Score | 108.6 | Prior+Duel hs |
| Video Games | Atari 2600 Tutankham | Score | 92.2 | DDQN (tuned) hs |
| Video Games | Atari 2600 Tutankham | Score | 68.1 | DQN noop |
| Video Games | Atari 2600 Tutankham | Score | 45.6 | DQN hs |
| Video Games | Atari 2600 Freeway | Score | 30.8 | DQN noop |
| Video Games | Atari 2600 Freeway | Score | 28.8 | DDQN (tuned) hs |
| Video Games | Atari 2600 Freeway | Score | 28.2 | Prior+Duel hs |
| Video Games | Atari 2600 Freeway | Score | 26.9 | DQN hs |
| Video Games | Atari 2600 Pong | Score | 19.5 | DQN noop |
| Video Games | Atari 2600 Pong | Score | 19.1 | DDQN (tuned) hs |
| Video Games | Atari 2600 Pong | Score | 18.4 | Prior+Duel hs |
| Video Games | Atari 2600 Pong | Score | 18 | DQN hs |
| Video Games | Atari 2600 Enduro | Score | 2223.9 | Prior+Duel hs |
| Video Games | Atari 2600 Enduro | Score | 1216.6 | DDQN (tuned) hs |
| Video Games | Atari 2600 Enduro | Score | 729 | DQN noop |
| Video Games | Atari 2600 Enduro | Score | 626.7 | DQN hs |
| Video Games | Atari 2600 Krull | Score | 8422.3 | DQN noop |
| Video Games | Atari 2600 Krull | Score | 7658.6 | Prior+Duel hs |
| Video Games | Atari 2600 Krull | Score | 6796.1 | DDQN (tuned) hs |
| Video Games | Atari 2600 Krull | Score | 6206 | DQN hs |
| Video Games | Atari 2600 Breakout | Score | 385.5 | DQN noop |
| Video Games | Atari 2600 Breakout | Score | 368.9 | DDQN (tuned) hs |
| Video Games | Atari 2600 Breakout | Score | 354.6 | Prior+Duel hs |
| Video Games | Atari 2600 Breakout | Score | 354.5 | DQN hs |
| Video Games | Atari 2600 Frostbite | Score | 4038.4 | Prior+Duel hs |
| Video Games | Atari 2600 Frostbite | Score | 1448.1 | DDQN (tuned) hs |
| Video Games | Atari 2600 Frostbite | Score | 797.4 | DQN noop |
| Video Games | Atari 2600 Frostbite | Score | 496.1 | DQN hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 47 | DQN hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 42 | DDQN (tuned) hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 24 | Prior+Duel hs |
| Video Games | Atari 2600 Gopher | Score | 105148.4 | Prior+Duel hs |
| Video Games | Atari 2600 Gopher | Score | 15253 | DDQN (tuned) hs |
| Video Games | Atari 2600 Gopher | Score | 8777.4 | DQN noop |
| Video Games | Atari 2600 Gopher | Score | 8190.4 | DQN hs |
| Video Games | Atari 2600 Space Invaders | Score | 8978 | Prior+Duel hs |
| Video Games | Atari 2600 Space Invaders | Score | 2628.7 | DDQN (tuned) hs |
| Video Games | Atari 2600 Space Invaders | Score | 1692.3 | DQN noop |
| Video Games | Atari 2600 Space Invaders | Score | 1293.8 | DQN hs |
| Video Games | Atari 2600 James Bond | Score | 768.5 | DQN noop |
| Video Games | Atari 2600 James Bond | Score | 697.5 | DQN hs |
| Video Games | Atari 2600 James Bond | Score | 585 | Prior+Duel hs |
| Video Games | Atari 2600 James Bond | Score | 573 | DDQN (tuned) hs |
| Video Games | Atari 2600 Amidar | Score | 978 | DQN noop |
| Video Games | Atari 2600 Amidar | Score | 178.4 | DQN hs |
| Video Games | Atari 2600 Amidar | Score | 169.1 | DDQN (tuned) hs |
| Video Games | Atari 2600 Tennis | Score | 12.2 | DQN noop |
| Video Games | Atari 2600 Tennis | Score | 11.1 | DQN hs |
| Video Games | Atari 2600 Tennis | Score | -7.8 | DDQN (tuned) hs |
| Video Games | Atari 2600 Tennis | Score | -13.2 | Prior+Duel hs |
| Video Games | Atari 2600 Crazy Climber | Score | 127853 | Prior+Duel hs |
| Video Games | Atari 2600 Crazy Climber | Score | 113782 | DDQN (tuned) hs |
| Video Games | Atari 2600 Crazy Climber | Score | 110763 | DQN noop |
| Video Games | Atari 2600 Crazy Climber | Score | 98128 | DQN hs |
| Video Games | Atari 2600 Asteroids | Score | 1458.7 | DQN hs |
| Video Games | Atari 2600 Asteroids | Score | 1364.5 | DQN noop |
| Video Games | Atari 2600 Asteroids | Score | 1193.2 | DDQN (tuned) hs |
| Video Games | Atari 2600 Asteroids | Score | 1021.9 | Prior+Duel hs |
| Video Games | Atari 2600 Gravitar | Score | 473 | DQN noop |
| Video Games | Atari 2600 Gravitar | Score | 298 | DQN hs |
| Video Games | Atari 2600 Gravitar | Score | 200.5 | DDQN (tuned) hs |
| Video Games | Atari 2600 Gravitar | Score | 167 | Prior+Duel hs |
| Video Games | Atari 2600 Time Pilot | Score | 6608 | DDQN (tuned) hs |
| Video Games | Atari 2600 Time Pilot | Score | 4871 | Prior+Duel hs |
| Video Games | Atari 2600 Time Pilot | Score | 4870 | DQN noop |
| Video Games | Atari 2600 Time Pilot | Score | 4786 | DQN hs |
| Video Games | Atari 2600 Demon Attack | Score | 73371.3 | Prior+Duel hs |
| Video Games | Atari 2600 Demon Attack | Score | 69803.4 | DDQN (tuned) hs |
| Video Games | Atari 2600 Demon Attack | Score | 12550.7 | DQN hs |
| Video Games | Atari 2600 Demon Attack | Score | 12149.4 | DQN noop |
| Video Games | Atari 2600 Battle Zone | Score | 30650 | Prior+Duel hs |
| Video Games | Atari 2600 Battle Zone | Score | 29900 | DQN noop |
| Video Games | Atari 2600 Battle Zone | Score | 24740 | DDQN (tuned) hs |
| Video Games | Atari 2600 Battle Zone | Score | 23750 | DQN hs |
| Video Games | Atari 2600 Beam Rider | Score | 37412.2 | Prior+Duel hs |
| Video Games | Atari 2600 Beam Rider | Score | 17417.2 | DDQN (tuned) hs |
| Video Games | Atari 2600 Beam Rider | Score | 9743.2 | DQN hs |
| Video Games | Atari 2600 Beam Rider | Score | 8627.5 | DQN noop |
| Video Games | Atari 2600 Asterix | Score | 16837 | DDQN (tuned) hs |
| Video Games | Atari 2600 Asterix | Score | 4359 | DQN noop |
| Video Games | Atari 2600 Asterix | Score | 3170.5 | DQN hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 37484 | Prior+Duel hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 30207 | DDQN (tuned) hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 26059 | DQN noop |
| Video Games | Atari 2600 Kung-Fu Master | Score | 20882 | DQN hs |
| Video Games | Atari 2600 Bowling | Score | 69.6 | DDQN (tuned) hs |
| Video Games | Atari 2600 Bowling | Score | 56.5 | DQN hs |
| Video Games | Atari 2600 Bowling | Score | 50.4 | DQN noop |
| Video Games | Atari 2600 Bowling | Score | 50.4 | Prior+Duel hs |
| Video Games | Atari 2600 Kangaroo | Score | 11204 | DDQN (tuned) hs |
| Video Games | Atari 2600 Kangaroo | Score | 7259 | DQN noop |
| Video Games | Atari 2600 Kangaroo | Score | 4496 | DQN hs |
| Video Games | Atari 2600 Kangaroo | Score | 861 | Prior+Duel hs |
| Video Games | Atari 2600 Assault | Score | 6060.8 | DDQN (tuned) hs |
| Video Games | Atari 2600 Assault | Score | 4280.4 | DQN noop |
| Video Games | Atari 2600 Assault | Score | 3489.3 | DQN hs |
| Video Games | Atari 2600 Alien | Score | 1620 | DQN noop |
| Video Games | Atari 2600 Alien | Score | 1033.4 | DDQN (tuned) hs |
| Video Games | Atari 2600 Alien | Score | 634 | DQN hs |
| Video Games | Atari 2600 Fishing Derby | Score | 17 | Prior+Duel hs |
| Video Games | Atari 2600 Fishing Derby | Score | 3.2 | DDQN (tuned) hs |
| Video Games | Atari 2600 Fishing Derby | Score | -1.6 | DQN hs |
| Video Games | Atari 2600 Fishing Derby | Score | -4.9 | DQN noop |
| Video Games | Atari 2600 Seaquest | Score | 14498 | DDQN (tuned) hs |
| Video Games | Atari 2600 Seaquest | Score | 5860.6 | DQN noop |
| Video Games | Atari 2600 Seaquest | Score | 4216.7 | DQN hs |
| Video Games | Atari 2600 Seaquest | Score | 1431.2 | Prior+Duel hs |
| Video Games | Atari 2600 Chopper Command | Score | 8058 | Prior+Duel hs |
| Video Games | Atari 2600 Chopper Command | Score | 6126 | DQN noop |
| Video Games | Atari 2600 Chopper Command | Score | 5017 | DQN hs |
| Video Games | Atari 2600 Chopper Command | Score | 3495 | DDQN (tuned) hs |
| Video Games | Atari 2600 Video Pinball | Score | 447408.6 | Prior+Duel hs |
| Video Games | Atari 2600 Video Pinball | Score | 367823.7 | DDQN (tuned) hs |
| Video Games | Atari 2600 Video Pinball | Score | 196760.4 | DQN noop |
| Video Games | Atari 2600 Video Pinball | Score | 154414.1 | DQN hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 10471 | Prior+Duel hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 6201 | DDQN (tuned) hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 2704 | DQN noop |
| Video Games | Atari 2600 Wizard of Wor | Score | 1609 | DQN hs |
| Video Games | Atari 2600 Zaxxon | Score | 11320 | Prior+Duel hs |
| Video Games | Atari 2600 Zaxxon | Score | 8593 | DDQN (tuned) hs |
| Video Games | Atari 2600 Zaxxon | Score | 5363 | DQN noop |
| Video Games | Atari 2600 Zaxxon | Score | 4412 | DQN hs |
| Video Games | Atari 2600 Robotank | Score | 63.9 | DQN noop |
| Video Games | Atari 2600 Robotank | Score | 59.1 | DDQN (tuned) hs |
| Video Games | Atari 2600 Robotank | Score | 58.7 | DQN hs |
| Video Games | Atari 2600 Robotank | Score | 24.7 | Prior+Duel hs |
| Video Games | Atari 2600 Name This Game | Score | 13637.9 | Prior+Duel hs |
| Video Games | Atari 2600 Name This Game | Score | 8960.3 | DDQN (tuned) hs |
| Video Games | Atari 2600 Name This Game | Score | 8207.8 | DQN noop |
| Video Games | Atari 2600 Name This Game | Score | 6738.8 | DQN hs |
| Video Games | Atari 2600 Star Gunner | Score | 127073 | Prior+Duel hs |
| Video Games | Atari 2600 Star Gunner | Score | 58365 | DDQN (tuned) hs |
| Video Games | Atari 2600 Star Gunner | Score | 54282 | DQN noop |
| Video Games | Atari 2600 Star Gunner | Score | 52970 | DQN hs |
| Video Games | Atari 2600 Ice Hockey | Score | 0.5 | Prior+Duel hs |
| Video Games | Atari 2600 Ice Hockey | Score | -1.6 | DQN hs |
| Video Games | Atari 2600 Ice Hockey | Score | -1.9 | DQN noop |
| Video Games | Atari 2600 Ice Hockey | Score | -2.5 | DDQN (tuned) hs |
| Video Games | Atari 2600 Berzerk | Score | 2178.6 | Prior+Duel hs |
| Video Games | Atari 2600 Berzerk | Score | 1011.1 | DDQN (tuned) hs |
| Video Games | Atari 2600 Berzerk | Score | 585.6 | DQN noop |
| Video Games | Atari 2600 Berzerk | Score | 493.4 | DQN hs |
| Video Games | Atari 2600 Atlantis | Score | 423252 | Prior+Duel hs |
| Video Games | Atari 2600 Atlantis | Score | 319688 | DDQN (tuned) hs |
| Video Games | Atari 2600 Atlantis | Score | 292491 | DQN hs |
| Video Games | Atari 2600 Atlantis | Score | 279987 | DQN noop |
| Video Games | Atari 2600 HERO | Score | 20437.8 | DQN noop |
| Video Games | Atari 2600 HERO | Score | 15459.2 | Prior+Duel hs |
| Video Games | Atari 2600 HERO | Score | 14992.9 | DQN hs |
| Video Games | Atari 2600 HERO | Score | 14892.5 | DDQN (tuned) hs |
| Video Games | Atari 2600 Bank Heist | Score | 1004.6 | Prior+Duel hs |
| Video Games | Atari 2600 Bank Heist | Score | 886 | DDQN (tuned) hs |
| Video Games | Atari 2600 Bank Heist | Score | 455 | DQN noop |
| Video Games | Atari 2600 Bank Heist | Score | 312.7 | DQN hs |
| Video Games | Atari 2600 Venture | Score | 163 | DQN noop |
| Video Games | Atari 2600 Venture | Score | 136 | DQN hs |
| Video Games | Atari 2600 Venture | Score | 29 | Prior+Duel hs |
| Video Games | Atari 2600 Venture | Score | 21 | DDQN (tuned) hs |
| Video Games | Atari 2600 Private Eye | Score | 1277.6 | Prior+Duel hs |
| Video Games | Atari 2600 Private Eye | Score | 207.9 | DQN hs |
| Video Games | Atari 2600 Private Eye | Score | 146.7 | DQN noop |
| Video Games | Atari 2600 Private Eye | Score | -575.5 | DDQN (tuned) hs |
| Video Games | Atari 2600 Q*Bert | Score | 14063 | Prior+Duel hs |
| Video Games | Atari 2600 Q*Bert | Score | 13117.3 | DQN noop |
| Video Games | Atari 2600 Q*Bert | Score | 11020.8 | DDQN (tuned) hs |
| Video Games | Atari 2600 Q*Bert | Score | 9271.5 | DQN hs |
| Video Games | Atari 2600 River Raid | Score | 16496.8 | Prior+Duel hs |
| Video Games | Atari 2600 River Raid | Score | 10838.4 | DDQN (tuned) hs |
| Video Games | Atari 2600 River Raid | Score | 7377.6 | DQN noop |
| Video Games | Atari 2600 River Raid | Score | 4748.5 | DQN hs |
| Video Games | Atari 2600 Road Runner | Score | 54630 | Prior+Duel hs |
| Video Games | Atari 2600 Road Runner | Score | 43156 | DDQN (tuned) hs |
| Video Games | Atari 2600 Road Runner | Score | 39544 | DQN noop |
| Video Games | Atari 2600 Road Runner | Score | 35215 | DQN hs |
| Video Games | Atari 2600 Up and Down | Score | 22681.3 | Prior+Duel hs |
| Video Games | Atari 2600 Up and Down | Score | 19086.9 | DDQN (tuned) hs |
| Video Games | Atari 2600 Up and Down | Score | 9989.9 | DQN noop |
| Video Games | Atari 2600 Up and Down | Score | 8038.5 | DQN hs |