Tom Schaul, John Quan, Ioannis Antonoglou, David Silver
Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Atari Games | Atari 2600 Boxing | Score | 95.6 | Prior noop |
| Atari Games | Atari 2600 Boxing | Score | 72.3 | Prior hs |
| Atari Games | Atari 2600 Double Dunk | Score | 18.5 | Prior noop |
| Atari Games | Atari 2600 Double Dunk | Score | 16 | Prior hs |
| Atari Games | Atari 2600 Ms. Pacman | Score | 6518.7 | Prior noop |
| Atari Games | Atari 2600 Ms. Pacman | Score | 1865.9 | Prior hs |
| Atari Games | Atari 2600 Centipede | Score | 4463.2 | Prior noop |
| Atari Games | Atari 2600 Centipede | Score | 3489.1 | Prior hs |
| Atari Games | Atari 2600 Tutankham | Score | 204.6 | Prior noop |
| Atari Games | Atari 2600 Tutankham | Score | 56.9 | Prior hs |
| Atari Games | Atari 2600 Freeway | Score | 33.7 | Prior noop |
| Atari Games | Atari 2600 Freeway | Score | 28.9 | Prior hs |
| Atari Games | Atari 2600 Pong | Score | 20.6 | Prior noop |
| Atari Games | Atari 2600 Pong | Score | 18.9 | Prior hs |
| Atari Games | Atari 2600 Enduro | Score | 2093 | Prior noop |
| Atari Games | Atari 2600 Enduro | Score | 1831 | Prior hs |
| Atari Games | Atari 2600 Krull | Score | 9728 | Prior noop |
| Atari Games | Atari 2600 Krull | Score | 6872.8 | Prior hs |
| Atari Games | Atari 2600 Breakout | Score | 373.9 | Prior noop |
| Atari Games | Atari 2600 Breakout | Score | 343 | Prior hs |
| Atari Games | Atari 2600 Frostbite | Score | 4380.1 | Prior noop |
| Atari Games | Atari 2600 Frostbite | Score | 3510 | Prior hs |
| Atari Games | Atari 2600 Montezuma's Revenge | Score | 51 | Prior hs |
| Atari Games | Atari 2600 Gopher | Score | 34858.8 | Prior hs |
| Atari Games | Atari 2600 Gopher | Score | 32487.2 | Prior noop |
| Atari Games | Atari 2600 Space Invaders | Score | 3912.1 | Prior hs |
| Atari Games | Atari 2600 Space Invaders | Score | 2865.8 | Prior noop |
| Atari Games | Atari 2600 James Bond | Score | 5148 | Prior noop |
| Atari Games | Atari 2600 James Bond | Score | 3961 | Prior hs |
| Atari Games | Atari 2600 Amidar | Score | 1838.9 | Prior noop |
| Atari Games | Atari 2600 Amidar | Score | 129.1 | Prior hs |
| Atari Games | Atari 2600 Tennis | Score | -5.3 | Prior hs |
| Atari Games | Atari 2600 Crazy Climber | Score | 141161 | Prior noop |
| Atari Games | Atari 2600 Crazy Climber | Score | 127512 | Prior hs |
| Atari Games | Atari 2600 Asteroids | Score | 2654.3 | Prior noop |
| Atari Games | Atari 2600 Asteroids | Score | 1745.1 | Prior hs |
| Atari Games | Atari 2600 Gravitar | Score | 548.5 | Prior noop |
| Atari Games | Atari 2600 Gravitar | Score | 269.5 | Prior hs |
| Atari Games | Atari 2600 Time Pilot | Score | 9197 | Prior noop |
| Atari Games | Atari 2600 Time Pilot | Score | 5963 | Prior hs |
| Atari Games | Atari 2600 Demon Attack | Score | 71846.4 | Prior noop |
| Atari Games | Atari 2600 Demon Attack | Score | 61277.5 | Prior hs |
| Atari Games | Atari 2600 Battle Zone | Score | 31530 | Prior noop |
| Atari Games | Atari 2600 Battle Zone | Score | 25520 | Prior hs |
| Atari Games | Atari 2600 Beam Rider | Score | 31181.3 | Prior hs |
| Atari Games | Atari 2600 Beam Rider | Score | 23384.2 | Prior noop |
| Atari Games | Atari 2600 Asterix | Score | 31527 | Prior noop |
| Atari Games | Atari 2600 Asterix | Score | 22484.5 | Prior hs |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 39581 | Prior noop |
| Atari Games | Atari 2600 Kung-Fu Master | Score | 31676 | Prior hs |
| Atari Games | Atari 2600 Bowling | Score | 52 | Prior hs |
| Atari Games | Atari 2600 Bowling | Score | 47.9 | Prior noop |
| Atari Games | Atari 2600 Kangaroo | Score | 16200 | Prior noop |
| Atari Games | Atari 2600 Kangaroo | Score | 12185 | Prior hs |
| Atari Games | Atari 2600 Assault | Score | 7672.1 | Prior noop |
| Atari Games | Atari 2600 Assault | Score | 6548.9 | Prior hs |
| Atari Games | Atari 2600 Alien | Score | 4203.8 | Prior noop |
| Atari Games | Atari 2600 Alien | Score | 1334.7 | Prior hs |
| Atari Games | Atari 2600 Fishing Derby | Score | 39.5 | Prior noop |
| Atari Games | Atari 2600 Fishing Derby | Score | 9.8 | Prior hs |
| Atari Games | Atari 2600 Seaquest | Score | 26357.8 | Prior noop |
| Atari Games | Atari 2600 Seaquest | Score | 25463.7 | Prior hs |
| Atari Games | Atari 2600 Chopper Command | Score | 8600 | Prior noop |
| Atari Games | Atari 2600 Chopper Command | Score | 4635 | Prior hs |
| Atari Games | Atari 2600 Video Pinball | Score | 295972.8 | Prior hs |
| Atari Games | Atari 2600 Video Pinball | Score | 282007.3 | Prior noop |
| Atari Games | Atari 2600 Wizard of Wor | Score | 5727 | Prior hs |
| Atari Games | Atari 2600 Wizard of Wor | Score | 4802 | Prior noop |
| Atari Games | Atari 2600 Zaxxon | Score | 10469 | Prior noop |
| Atari Games | Atari 2600 Zaxxon | Score | 9474 | Prior hs |
| Atari Games | Atari 2600 Robotank | Score | 62.6 | Prior noop |
| Atari Games | Atari 2600 Robotank | Score | 56.2 | Prior hs |
| Atari Games | Atari 2600 Name This Game | Score | 12270.5 | Prior noop |
| Atari Games | Atari 2600 Name This Game | Score | 10497.6 | Prior hs |
| Atari Games | Atari 2600 Star Gunner | Score | 63302 | Prior noop |
| Atari Games | Atari 2600 Star Gunner | Score | 61582 | Prior hs |
| Atari Games | Atari 2600 Ice Hockey | Score | 1.3 | Prior noop |
| Atari Games | Atari 2600 Ice Hockey | Score | -0.2 | Prior hs |
| Atari Games | Atari 2600 Berzerk | Score | 1305.6 | Prior noop |
| Atari Games | Atari 2600 Berzerk | Score | 865.9 | Prior hs |
| Atari Games | Atari 2600 Atlantis | Score | 357324 | Prior noop |
| Atari Games | Atari 2600 Atlantis | Score | 330647 | Prior hs |
| Atari Games | Atari 2600 HERO | Score | 23037.7 | Prior noop |
| Atari Games | Atari 2600 HERO | Score | 20889.9 | Prior hs |
| Atari Games | Atari 2600 Bank Heist | Score | 1054.6 | Prior noop |
| Atari Games | Atari 2600 Bank Heist | Score | 876.6 | Prior hs |
| Atari Games | Atari 2600 Venture | Score | 94 | Prior hs |
| Atari Games | Atari 2600 Venture | Score | 54 | Prior noop |
| Atari Games | Atari 2600 Private Eye | Score | 670.7 | Prior hs |
| Atari Games | Atari 2600 Private Eye | Score | 200 | Prior noop |
| Atari Games | Atari 2600 Q*Bert | Score | 16256.5 | Prior noop |
| Atari Games | Atari 2600 Q*Bert | Score | 9944 | Prior hs |
| Atari Games | Atari 2600 River Raid | Score | 14522.3 | Prior noop |
| Atari Games | Atari 2600 River Raid | Score | 11807.2 | Prior hs |
| Atari Games | Atari 2600 Road Runner | Score | 57608 | Prior noop |
| Atari Games | Atari 2600 Road Runner | Score | 52264 | Prior hs |
| Atari Games | Atari 2600 Up and Down | Score | 16154.1 | Prior noop |
| Atari Games | Atari 2600 Up and Down | Score | 12157.4 | Prior hs |
| Video Games | Atari 2600 Boxing | Score | 95.6 | Prior noop |
| Video Games | Atari 2600 Boxing | Score | 72.3 | Prior hs |
| Video Games | Atari 2600 Double Dunk | Score | 18.5 | Prior noop |
| Video Games | Atari 2600 Double Dunk | Score | 16 | Prior hs |
| Video Games | Atari 2600 Ms. Pacman | Score | 6518.7 | Prior noop |
| Video Games | Atari 2600 Ms. Pacman | Score | 1865.9 | Prior hs |
| Video Games | Atari 2600 Centipede | Score | 4463.2 | Prior noop |
| Video Games | Atari 2600 Centipede | Score | 3489.1 | Prior hs |
| Video Games | Atari 2600 Tutankham | Score | 204.6 | Prior noop |
| Video Games | Atari 2600 Tutankham | Score | 56.9 | Prior hs |
| Video Games | Atari 2600 Freeway | Score | 33.7 | Prior noop |
| Video Games | Atari 2600 Freeway | Score | 28.9 | Prior hs |
| Video Games | Atari 2600 Pong | Score | 20.6 | Prior noop |
| Video Games | Atari 2600 Pong | Score | 18.9 | Prior hs |
| Video Games | Atari 2600 Enduro | Score | 2093 | Prior noop |
| Video Games | Atari 2600 Enduro | Score | 1831 | Prior hs |
| Video Games | Atari 2600 Krull | Score | 9728 | Prior noop |
| Video Games | Atari 2600 Krull | Score | 6872.8 | Prior hs |
| Video Games | Atari 2600 Breakout | Score | 373.9 | Prior noop |
| Video Games | Atari 2600 Breakout | Score | 343 | Prior hs |
| Video Games | Atari 2600 Frostbite | Score | 4380.1 | Prior noop |
| Video Games | Atari 2600 Frostbite | Score | 3510 | Prior hs |
| Video Games | Atari 2600 Montezuma's Revenge | Score | 51 | Prior hs |
| Video Games | Atari 2600 Gopher | Score | 34858.8 | Prior hs |
| Video Games | Atari 2600 Gopher | Score | 32487.2 | Prior noop |
| Video Games | Atari 2600 Space Invaders | Score | 3912.1 | Prior hs |
| Video Games | Atari 2600 Space Invaders | Score | 2865.8 | Prior noop |
| Video Games | Atari 2600 James Bond | Score | 5148 | Prior noop |
| Video Games | Atari 2600 James Bond | Score | 3961 | Prior hs |
| Video Games | Atari 2600 Amidar | Score | 1838.9 | Prior noop |
| Video Games | Atari 2600 Amidar | Score | 129.1 | Prior hs |
| Video Games | Atari 2600 Tennis | Score | -5.3 | Prior hs |
| Video Games | Atari 2600 Crazy Climber | Score | 141161 | Prior noop |
| Video Games | Atari 2600 Crazy Climber | Score | 127512 | Prior hs |
| Video Games | Atari 2600 Asteroids | Score | 2654.3 | Prior noop |
| Video Games | Atari 2600 Asteroids | Score | 1745.1 | Prior hs |
| Video Games | Atari 2600 Gravitar | Score | 548.5 | Prior noop |
| Video Games | Atari 2600 Gravitar | Score | 269.5 | Prior hs |
| Video Games | Atari 2600 Time Pilot | Score | 9197 | Prior noop |
| Video Games | Atari 2600 Time Pilot | Score | 5963 | Prior hs |
| Video Games | Atari 2600 Demon Attack | Score | 71846.4 | Prior noop |
| Video Games | Atari 2600 Demon Attack | Score | 61277.5 | Prior hs |
| Video Games | Atari 2600 Battle Zone | Score | 31530 | Prior noop |
| Video Games | Atari 2600 Battle Zone | Score | 25520 | Prior hs |
| Video Games | Atari 2600 Beam Rider | Score | 31181.3 | Prior hs |
| Video Games | Atari 2600 Beam Rider | Score | 23384.2 | Prior noop |
| Video Games | Atari 2600 Asterix | Score | 31527 | Prior noop |
| Video Games | Atari 2600 Asterix | Score | 22484.5 | Prior hs |
| Video Games | Atari 2600 Kung-Fu Master | Score | 39581 | Prior noop |
| Video Games | Atari 2600 Kung-Fu Master | Score | 31676 | Prior hs |
| Video Games | Atari 2600 Bowling | Score | 52 | Prior hs |
| Video Games | Atari 2600 Bowling | Score | 47.9 | Prior noop |
| Video Games | Atari 2600 Kangaroo | Score | 16200 | Prior noop |
| Video Games | Atari 2600 Kangaroo | Score | 12185 | Prior hs |
| Video Games | Atari 2600 Assault | Score | 7672.1 | Prior noop |
| Video Games | Atari 2600 Assault | Score | 6548.9 | Prior hs |
| Video Games | Atari 2600 Alien | Score | 4203.8 | Prior noop |
| Video Games | Atari 2600 Alien | Score | 1334.7 | Prior hs |
| Video Games | Atari 2600 Fishing Derby | Score | 39.5 | Prior noop |
| Video Games | Atari 2600 Fishing Derby | Score | 9.8 | Prior hs |
| Video Games | Atari 2600 Seaquest | Score | 26357.8 | Prior noop |
| Video Games | Atari 2600 Seaquest | Score | 25463.7 | Prior hs |
| Video Games | Atari 2600 Chopper Command | Score | 8600 | Prior noop |
| Video Games | Atari 2600 Chopper Command | Score | 4635 | Prior hs |
| Video Games | Atari 2600 Video Pinball | Score | 295972.8 | Prior hs |
| Video Games | Atari 2600 Video Pinball | Score | 282007.3 | Prior noop |
| Video Games | Atari 2600 Wizard of Wor | Score | 5727 | Prior hs |
| Video Games | Atari 2600 Wizard of Wor | Score | 4802 | Prior noop |
| Video Games | Atari 2600 Zaxxon | Score | 10469 | Prior noop |
| Video Games | Atari 2600 Zaxxon | Score | 9474 | Prior hs |
| Video Games | Atari 2600 Robotank | Score | 62.6 | Prior noop |
| Video Games | Atari 2600 Robotank | Score | 56.2 | Prior hs |
| Video Games | Atari 2600 Name This Game | Score | 12270.5 | Prior noop |
| Video Games | Atari 2600 Name This Game | Score | 10497.6 | Prior hs |
| Video Games | Atari 2600 Star Gunner | Score | 63302 | Prior noop |
| Video Games | Atari 2600 Star Gunner | Score | 61582 | Prior hs |
| Video Games | Atari 2600 Ice Hockey | Score | 1.3 | Prior noop |
| Video Games | Atari 2600 Ice Hockey | Score | -0.2 | Prior hs |
| Video Games | Atari 2600 Berzerk | Score | 1305.6 | Prior noop |
| Video Games | Atari 2600 Berzerk | Score | 865.9 | Prior hs |
| Video Games | Atari 2600 Atlantis | Score | 357324 | Prior noop |
| Video Games | Atari 2600 Atlantis | Score | 330647 | Prior hs |
| Video Games | Atari 2600 HERO | Score | 23037.7 | Prior noop |
| Video Games | Atari 2600 HERO | Score | 20889.9 | Prior hs |
| Video Games | Atari 2600 Bank Heist | Score | 1054.6 | Prior noop |
| Video Games | Atari 2600 Bank Heist | Score | 876.6 | Prior hs |
| Video Games | Atari 2600 Venture | Score | 94 | Prior hs |
| Video Games | Atari 2600 Venture | Score | 54 | Prior noop |
| Video Games | Atari 2600 Private Eye | Score | 670.7 | Prior hs |
| Video Games | Atari 2600 Private Eye | Score | 200 | Prior noop |
| Video Games | Atari 2600 Q*Bert | Score | 16256.5 | Prior noop |
| Video Games | Atari 2600 Q*Bert | Score | 9944 | Prior hs |
| Video Games | Atari 2600 River Raid | Score | 14522.3 | Prior noop |
| Video Games | Atari 2600 River Raid | Score | 11807.2 | Prior hs |
| Video Games | Atari 2600 Road Runner | Score | 57608 | Prior noop |
| Video Games | Atari 2600 Road Runner | Score | 52264 | Prior hs |
| Video Games | Atari 2600 Up and Down | Score | 16154.1 | Prior noop |
| Video Games | Atari 2600 Up and Down | Score | 12157.4 | Prior hs |