Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Atari Games | Atari 2600 Boxing | Score | 59.8 | A3C FF hs |
| Atari Games | Atari 2600 Boxing | Score | 37.3 | A3C LSTM hs |
| Atari Games | Atari 2600 Boxing | Score | 33.7 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Double Dunk | Score | 0.1 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Double Dunk | Score | 0.1 | A3C LSTM hs |
| Atari Games | Atari 2600 Double Dunk | Score | -0.1 | A3C FF hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 850.7 | A3C LSTM hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 653.7 | A3C FF hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 594.4 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Centipede | Score | 3755.8 | A3C FF hs |
| Atari Games | Atari 2600 Centipede | Score | 3306.5 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Centipede | Score | 1997 | A3C LSTM hs |
| Atari Games | Atari 2600 Tutankham | Score | 156.3 | A3C FF hs |
| Atari Games | Atari 2600 Tutankham | Score | 144.2 | A3C LSTM hs |
| Atari Games | Atari 2600 Tutankham | Score | 26.1 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Freeway | Score | 0.1 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Freeway | Score | 0.1 | A3C FF hs |
| Atari Games | Atari 2600 Freeway | Score | 0.1 | A3C LSTM hs |
| Atari Games | Atari 2600 Pong | Score | 11.4 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Pong | Score | 10.7 | A3C LSTM hs |
| Atari Games | Atari 2600 Pong | Score | 5.6 | A3C FF hs |
| Atari Games | Atari 2600 Enduro | Score | -82.2 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Enduro | Score | -82.5 | A3C FF hs |
| Atari Games | Atari 2600 Enduro | Score | -82.5 | A3C LSTM hs |
| Atari Games | Atari 2600 Krull | Score | 8066.6 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Krull | Score | 5911.4 | A3C LSTM hs |
| Atari Games | Atari 2600 Krull | Score | 5560 | A3C FF hs |
| Atari Games | Atari 2600 Breakout | Score | 766.8 | A3C LSTM hs |
| Atari Games | Atari 2600 Breakout | Score | 681.9 | A3C FF hs |
| Atari Games | Atari 2600 Breakout | Score | 551.6 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Frostbite | Score | 197.6 | A3C LSTM hs |
| Atari Games | Atari 2600 Frostbite | Score | 190.5 | A3C FF hs |
| Atari Games | Atari 2600 Frostbite | Score | 180.1 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 67 | A3C FF hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 53 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 41 | A3C LSTM hs |
| Atari Games | Atari 2600 Gopher | Score | 17106.8 | A3C LSTM hs |
| Atari Games | Atari 2600 Gopher | Score | 10022.8 | A3C FF hs |
| Atari Games | Atari 2600 Gopher | Score | 8442.8 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Space Invaders | Score | 23846 | A3C LSTM hs |
| Atari Games | Atari 2600 Space Invaders | Score | 15730.5 | A3C FF hs |
| Atari Games | Atari 2600 Space Invaders | Score | 2214.7 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 James Bond | Score | 613 | A3C LSTM hs |
| Atari Games | Atari 2600 James Bond | Score | 541 | A3C FF hs |
| Atari Games | Atari 2600 James Bond | Score | 351.5 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Amidar | Score | 283.9 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Amidar | Score | 263.9 | A3C FF hs |
| Atari Games | Atari 2600 Amidar | Score | 173 | A3C LSTM hs |
| Atari Games | Atari 2600 Tennis | Score | -6.3 | A3C FF hs |
| Atari Games | Atari 2600 Tennis | Score | -6.4 | A3C LSTM hs |
| Atari Games | Atari 2600 Tennis | Score | -10.2 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 138518 | A3C LSTM hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 112646 | A3C FF hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 101624 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Asteroids | Score | 5093.1 | A3C LSTM hs |
| Atari Games | Atari 2600 Asteroids | Score | 4474.5 | A3C FF hs |
| Atari Games | Atari 2600 Asteroids | Score | 3009.4 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Gravitar | Score | 320 | A3C LSTM hs |
| Atari Games | Atari 2600 Gravitar | Score | 303.5 | A3C FF hs |
| Atari Games | Atari 2600 Gravitar | Score | 269.5 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Time Pilot | Score | 27202 | A3C LSTM hs |
| Atari Games | Atari 2600 Time Pilot | Score | 12679 | A3C FF hs |
| Atari Games | Atari 2600 Time Pilot | Score | 5825 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Demon Attack | Score | 115201.9 | A3C LSTM hs |
| Atari Games | Atari 2600 Demon Attack | Score | 113308.4 | A3C FF hs |
| Atari Games | Atari 2600 Demon Attack | Score | 84997.5 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Battle Zone | Score | 20760 | A3C LSTM hs |
| Atari Games | Atari 2600 Battle Zone | Score | 12950 | A3C FF hs |
| Atari Games | Atari 2600 Battle Zone | Score | 11340 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Beam Rider | Score | 24622.2 | A3C LSTM hs |
| Atari Games | Atari 2600 Beam Rider | Score | 22707.9 | A3C FF hs |
| Atari Games | Atari 2600 Beam Rider | Score | 13235.9 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Asterix | Score | 22140.5 | A3C FF hs |
| Atari Games | Atari 2600 Asterix | Score | 17244.5 | A3C LSTM hs |
| Atari Games | Atari 2600 Asterix | Score | 6723 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 40835 | A3C LSTM hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 28819 | A3C FF hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 3046 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Bowling | Score | 41.8 | A3C LSTM hs |
| Atari Games | Atari 2600 Bowling | Score | 36.2 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Bowling | Score | 35.1 | A3C FF hs |
| Atari Games | Atari 2600 Kangaroo | Score | 125 | A3C LSTM hs |
| Atari Games | Atari 2600 Kangaroo | Score | 106 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Kangaroo | Score | 94 | A3C FF hs |
| Atari Games | Atari 2600 Assault | Score | 14497.9 | A3C LSTM hs |
| Atari Games | Atari 2600 Assault | Score | 5474.9 | A3C FF hs |
| Atari Games | Atari 2600 Assault | Score | 3746.1 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Alien | Score | 945.3 | A3C LSTM hs |
| Atari Games | Atari 2600 Alien | Score | 518.4 | A3C FF hs |
| Atari Games | Atari 2600 Alien | Score | 182.1 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Fishing Derby | Score | 22.6 | A3C LSTM hs |
| Atari Games | Atari 2600 Fishing Derby | Score | 18.8 | A3C FF hs |
| Atari Games | Atari 2600 Fishing Derby | Score | 13.6 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Seaquest | Score | 2355.4 | A3C FF hs |
| Atari Games | Atari 2600 Seaquest | Score | 2300.2 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Seaquest | Score | 1326.1 | A3C LSTM hs |
| Atari Games | Atari 2600 Chopper Command | Score | 10150 | A3C LSTM hs |
| Atari Games | Atari 2600 Chopper Command | Score | 7021 | A3C FF hs |
| Atari Games | Atari 2600 Chopper Command | Score | 4669 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Video Pinball | Score | 470310.5 | A3C LSTM hs |
| Atari Games | Atari 2600 Video Pinball | Score | 331628.1 | A3C FF hs |
| Atari Games | Atari 2600 Video Pinball | Score | 185852.6 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 18082 | A3C LSTM hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 17244 | A3C FF hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 5278 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Zaxxon | Score | 24622 | A3C FF hs |
| Atari Games | Atari 2600 Zaxxon | Score | 23519 | A3C LSTM hs |
| Atari Games | Atari 2600 Zaxxon | Score | 2659 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Robotank | Score | 32.8 | A3C FF hs |
| Atari Games | Atari 2600 Robotank | Score | 2.6 | A3C LSTM hs |
| Atari Games | Atari 2600 Robotank | Score | 2.3 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Name This Game | Score | 12093.7 | A3C LSTM hs |
| Atari Games | Atari 2600 Name This Game | Score | 10476.1 | A3C FF hs |
| Atari Games | Atari 2600 Name This Game | Score | 5614 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Star Gunner | Score | 164766 | A3C LSTM hs |
| Atari Games | Atari 2600 Star Gunner | Score | 138218 | A3C FF hs |
| Atari Games | Atari 2600 Star Gunner | Score | 64393 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Ice Hockey | Score | -1.7 | A3C LSTM hs |
| Atari Games | Atari 2600 Ice Hockey | Score | -2.8 | A3C FF hs |
| Atari Games | Atari 2600 Ice Hockey | Score | -4.7 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Berzerk | Score | 1433.4 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Berzerk | Score | 862.2 | A3C LSTM hs |
| Atari Games | Atari 2600 Berzerk | Score | 817.9 | A3C FF hs |
| Atari Games | Atari 2600 Atlantis | Score | 911091 | A3C FF hs |
| Atari Games | Atari 2600 Atlantis | Score | 875822 | A3C LSTM hs |
| Atari Games | Atari 2600 Atlantis | Score | 772392 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 HERO | Score | 32464.1 | A3C FF hs |
| Atari Games | Atari 2600 HERO | Score | 28889.5 | A3C LSTM hs |
| Atari Games | Atari 2600 HERO | Score | 28765.8 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Bank Heist | Score | 970.1 | A3C FF hs |
| Atari Games | Atari 2600 Bank Heist | Score | 946 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Bank Heist | Score | 932.8 | A3C LSTM hs |
| Atari Games | Atari 2600 Venture | Score | 25 | A3C LSTM hs |
| Atari Games | Atari 2600 Venture | Score | 23 | A3C FF hs |
| Atari Games | Atari 2600 Venture | Score | 19 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Private Eye | Score | 421.1 | A3C LSTM hs |
| Atari Games | Atari 2600 Private Eye | Score | 206.9 | A3C FF hs |
| Atari Games | Atari 2600 Private Eye | Score | 194.4 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Q*Bert | Score | 21307.5 | A3C LSTM hs |
| Atari Games | Atari 2600 Q*Bert | Score | 15148.8 | A3C FF hs |
| Atari Games | Atari 2600 Q*Bert | Score | 13752.3 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 River Raid | Score | 12201.8 | A3C FF hs |
| Atari Games | Atari 2600 River Raid | Score | 10001.2 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 River Raid | Score | 6591.9 | A3C LSTM hs |
| Atari Games | Atari 2600 Road Runner | Score | 73949 | A3C LSTM hs |
| Atari Games | Atari 2600 Road Runner | Score | 34216 | A3C FF hs |
| Atari Games | Atari 2600 Road Runner | Score | 31769 | A3C FF (1 day) hs |
| Atari Games | Atari 2600 Up and Down | Score | 105728.7 | A3C LSTM hs |
| Atari Games | Atari 2600 Up and Down | Score | 74705.7 | A3C FF hs |
| Atari Games | Atari 2600 Up and Down | Score | 54525.4 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Boxing | Score | 59.8 | A3C FF hs |
| Video Games | Atari 2600 Boxing | Score | 37.3 | A3C LSTM hs |
| Video Games | Atari 2600 Boxing | Score | 33.7 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Double Dunk | Score | 0.1 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Double Dunk | Score | 0.1 | A3C LSTM hs |
| Video Games | Atari 2600 Double Dunk | Score | -0.1 | A3C FF hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 850.7 | A3C LSTM hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 653.7 | A3C FF hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 594.4 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Centipede | Score | 3755.8 | A3C FF hs |
| Video Games | Atari 2600 Centipede | Score | 3306.5 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Centipede | Score | 1997 | A3C LSTM hs |
| Video Games | Atari 2600 Tutankham | Score | 156.3 | A3C FF hs |
| Video Games | Atari 2600 Tutankham | Score | 144.2 | A3C LSTM hs |
| Video Games | Atari 2600 Tutankham | Score | 26.1 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Freeway | Score | 0.1 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Freeway | Score | 0.1 | A3C FF hs |
| Video Games | Atari 2600 Freeway | Score | 0.1 | A3C LSTM hs |
| Video Games | Atari 2600 Pong | Score | 11.4 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Pong | Score | 10.7 | A3C LSTM hs |
| Video Games | Atari 2600 Pong | Score | 5.6 | A3C FF hs |
| Video Games | Atari 2600 Enduro | Score | -82.2 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Enduro | Score | -82.5 | A3C FF hs |
| Video Games | Atari 2600 Enduro | Score | -82.5 | A3C LSTM hs |
| Video Games | Atari 2600 Krull | Score | 8066.6 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Krull | Score | 5911.4 | A3C LSTM hs |
| Video Games | Atari 2600 Krull | Score | 5560 | A3C FF hs |
| Video Games | Atari 2600 Breakout | Score | 766.8 | A3C LSTM hs |
| Video Games | Atari 2600 Breakout | Score | 681.9 | A3C FF hs |
| Video Games | Atari 2600 Breakout | Score | 551.6 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Frostbite | Score | 197.6 | A3C LSTM hs |
| Video Games | Atari 2600 Frostbite | Score | 190.5 | A3C FF hs |
| Video Games | Atari 2600 Frostbite | Score | 180.1 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 67 | A3C FF hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 53 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 41 | A3C LSTM hs |
| Video Games | Atari 2600 Gopher | Score | 17106.8 | A3C LSTM hs |
| Video Games | Atari 2600 Gopher | Score | 10022.8 | A3C FF hs |
| Video Games | Atari 2600 Gopher | Score | 8442.8 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Space Invaders | Score | 23846 | A3C LSTM hs |
| Video Games | Atari 2600 Space Invaders | Score | 15730.5 | A3C FF hs |
| Video Games | Atari 2600 Space Invaders | Score | 2214.7 | A3C FF (1 day) hs |
| Video Games | Atari 2600 James Bond | Score | 613 | A3C LSTM hs |
| Video Games | Atari 2600 James Bond | Score | 541 | A3C FF hs |
| Video Games | Atari 2600 James Bond | Score | 351.5 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Amidar | Score | 283.9 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Amidar | Score | 263.9 | A3C FF hs |
| Video Games | Atari 2600 Amidar | Score | 173 | A3C LSTM hs |
| Video Games | Atari 2600 Tennis | Score | -6.3 | A3C FF hs |
| Video Games | Atari 2600 Tennis | Score | -6.4 | A3C LSTM hs |
| Video Games | Atari 2600 Tennis | Score | -10.2 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Crazy Climber | Score | 138518 | A3C LSTM hs |
| Video Games | Atari 2600 Crazy Climber | Score | 112646 | A3C FF hs |
| Video Games | Atari 2600 Crazy Climber | Score | 101624 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Asteroids | Score | 5093.1 | A3C LSTM hs |
| Video Games | Atari 2600 Asteroids | Score | 4474.5 | A3C FF hs |
| Video Games | Atari 2600 Asteroids | Score | 3009.4 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Gravitar | Score | 320 | A3C LSTM hs |
| Video Games | Atari 2600 Gravitar | Score | 303.5 | A3C FF hs |
| Video Games | Atari 2600 Gravitar | Score | 269.5 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Time Pilot | Score | 27202 | A3C LSTM hs |
| Video Games | Atari 2600 Time Pilot | Score | 12679 | A3C FF hs |
| Video Games | Atari 2600 Time Pilot | Score | 5825 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Demon Attack | Score | 115201.9 | A3C LSTM hs |
| Video Games | Atari 2600 Demon Attack | Score | 113308.4 | A3C FF hs |
| Video Games | Atari 2600 Demon Attack | Score | 84997.5 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Battle Zone | Score | 20760 | A3C LSTM hs |
| Video Games | Atari 2600 Battle Zone | Score | 12950 | A3C FF hs |
| Video Games | Atari 2600 Battle Zone | Score | 11340 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Beam Rider | Score | 24622.2 | A3C LSTM hs |
| Video Games | Atari 2600 Beam Rider | Score | 22707.9 | A3C FF hs |
| Video Games | Atari 2600 Beam Rider | Score | 13235.9 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Asterix | Score | 22140.5 | A3C FF hs |
| Video Games | Atari 2600 Asterix | Score | 17244.5 | A3C LSTM hs |
| Video Games | Atari 2600 Asterix | Score | 6723 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 40835 | A3C LSTM hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 28819 | A3C FF hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 3046 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Bowling | Score | 41.8 | A3C LSTM hs |
| Video Games | Atari 2600 Bowling | Score | 36.2 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Bowling | Score | 35.1 | A3C FF hs |
| Video Games | Atari 2600 Kangaroo | Score | 125 | A3C LSTM hs |
| Video Games | Atari 2600 Kangaroo | Score | 106 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Kangaroo | Score | 94 | A3C FF hs |
| Video Games | Atari 2600 Assault | Score | 14497.9 | A3C LSTM hs |
| Video Games | Atari 2600 Assault | Score | 5474.9 | A3C FF hs |
| Video Games | Atari 2600 Assault | Score | 3746.1 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Alien | Score | 945.3 | A3C LSTM hs |
| Video Games | Atari 2600 Alien | Score | 518.4 | A3C FF hs |
| Video Games | Atari 2600 Alien | Score | 182.1 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Fishing Derby | Score | 22.6 | A3C LSTM hs |
| Video Games | Atari 2600 Fishing Derby | Score | 18.8 | A3C FF hs |
| Video Games | Atari 2600 Fishing Derby | Score | 13.6 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Seaquest | Score | 2355.4 | A3C FF hs |
| Video Games | Atari 2600 Seaquest | Score | 2300.2 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Seaquest | Score | 1326.1 | A3C LSTM hs |
| Video Games | Atari 2600 Chopper Command | Score | 10150 | A3C LSTM hs |
| Video Games | Atari 2600 Chopper Command | Score | 7021 | A3C FF hs |
| Video Games | Atari 2600 Chopper Command | Score | 4669 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Video Pinball | Score | 470310.5 | A3C LSTM hs |
| Video Games | Atari 2600 Video Pinball | Score | 331628.1 | A3C FF hs |
| Video Games | Atari 2600 Video Pinball | Score | 185852.6 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 18082 | A3C LSTM hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 17244 | A3C FF hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 5278 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Zaxxon | Score | 24622 | A3C FF hs |
| Video Games | Atari 2600 Zaxxon | Score | 23519 | A3C LSTM hs |
| Video Games | Atari 2600 Zaxxon | Score | 2659 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Robotank | Score | 32.8 | A3C FF hs |
| Video Games | Atari 2600 Robotank | Score | 2.6 | A3C LSTM hs |
| Video Games | Atari 2600 Robotank | Score | 2.3 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Name This Game | Score | 12093.7 | A3C LSTM hs |
| Video Games | Atari 2600 Name This Game | Score | 10476.1 | A3C FF hs |
| Video Games | Atari 2600 Name This Game | Score | 5614 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Star Gunner | Score | 164766 | A3C LSTM hs |
| Video Games | Atari 2600 Star Gunner | Score | 138218 | A3C FF hs |
| Video Games | Atari 2600 Star Gunner | Score | 64393 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Ice Hockey | Score | -1.7 | A3C LSTM hs |
| Video Games | Atari 2600 Ice Hockey | Score | -2.8 | A3C FF hs |
| Video Games | Atari 2600 Ice Hockey | Score | -4.7 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Berzerk | Score | 1433.4 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Berzerk | Score | 862.2 | A3C LSTM hs |
| Video Games | Atari 2600 Berzerk | Score | 817.9 | A3C FF hs |
| Video Games | Atari 2600 Atlantis | Score | 911091 | A3C FF hs |
| Video Games | Atari 2600 Atlantis | Score | 875822 | A3C LSTM hs |
| Video Games | Atari 2600 Atlantis | Score | 772392 | A3C FF (1 day) hs |
| Video Games | Atari 2600 HERO | Score | 32464.1 | A3C FF hs |
| Video Games | Atari 2600 HERO | Score | 28889.5 | A3C LSTM hs |
| Video Games | Atari 2600 HERO | Score | 28765.8 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Bank Heist | Score | 970.1 | A3C FF hs |
| Video Games | Atari 2600 Bank Heist | Score | 946 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Bank Heist | Score | 932.8 | A3C LSTM hs |
| Video Games | Atari 2600 Venture | Score | 25 | A3C LSTM hs |
| Video Games | Atari 2600 Venture | Score | 23 | A3C FF hs |
| Video Games | Atari 2600 Venture | Score | 19 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Private Eye | Score | 421.1 | A3C LSTM hs |
| Video Games | Atari 2600 Private Eye | Score | 206.9 | A3C FF hs |
| Video Games | Atari 2600 Private Eye | Score | 194.4 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Q*Bert | Score | 21307.5 | A3C LSTM hs |
| Video Games | Atari 2600 Q*Bert | Score | 15148.8 | A3C FF hs |
| Video Games | Atari 2600 Q*Bert | Score | 13752.3 | A3C FF (1 day) hs |
| Video Games | Atari 2600 River Raid | Score | 12201.8 | A3C FF hs |
| Video Games | Atari 2600 River Raid | Score | 10001.2 | A3C FF (1 day) hs |
| Video Games | Atari 2600 River Raid | Score | 6591.9 | A3C LSTM hs |
| Video Games | Atari 2600 Road Runner | Score | 73949 | A3C LSTM hs |
| Video Games | Atari 2600 Road Runner | Score | 34216 | A3C FF hs |
| Video Games | Atari 2600 Road Runner | Score | 31769 | A3C FF (1 day) hs |
| Video Games | Atari 2600 Up and Down | Score | 105728.7 | A3C LSTM hs |
| Video Games | Atari 2600 Up and Down | Score | 74705.7 | A3C FF hs |
| Video Games | Atari 2600 Up and Down | Score | 54525.4 | A3C FF (1 day) hs |