TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Papers/Prioritized Experience Replay

Prioritized Experience Replay

Tom Schaul, John Quan, Ioannis Antonoglou, David Silver

2015-11-18Reinforcement LearningAtari Gamesreinforcement-learning
PaperPDFCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCode

Abstract

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.

Results

TaskDatasetMetricValueModel
Atari GamesAtari 2600 BoxingScore95.6Prior noop
Atari GamesAtari 2600 BoxingScore72.3Prior hs
Atari GamesAtari 2600 Double DunkScore18.5Prior noop
Atari GamesAtari 2600 Double DunkScore16Prior hs
Atari GamesAtari 2600 Ms. PacmanScore6518.7Prior noop
Atari GamesAtari 2600 Ms. PacmanScore1865.9Prior hs
Atari GamesAtari 2600 CentipedeScore4463.2Prior noop
Atari GamesAtari 2600 CentipedeScore3489.1Prior hs
Atari GamesAtari 2600 TutankhamScore204.6Prior noop
Atari GamesAtari 2600 TutankhamScore56.9Prior hs
Atari GamesAtari 2600 FreewayScore33.7Prior noop
Atari GamesAtari 2600 FreewayScore28.9Prior hs
Atari GamesAtari 2600 PongScore20.6Prior noop
Atari GamesAtari 2600 PongScore18.9Prior hs
Atari GamesAtari 2600 EnduroScore2093Prior noop
Atari GamesAtari 2600 EnduroScore1831Prior hs
Atari GamesAtari 2600 KrullScore9728Prior noop
Atari GamesAtari 2600 KrullScore6872.8Prior hs
Atari GamesAtari 2600 BreakoutScore373.9Prior noop
Atari GamesAtari 2600 BreakoutScore343Prior hs
Atari GamesAtari 2600 FrostbiteScore4380.1Prior noop
Atari GamesAtari 2600 FrostbiteScore3510Prior hs
Atari GamesAtari 2600 Montezuma's RevengeScore51Prior hs
Atari GamesAtari 2600 GopherScore34858.8Prior hs
Atari GamesAtari 2600 GopherScore32487.2Prior noop
Atari GamesAtari 2600 Space InvadersScore3912.1Prior hs
Atari GamesAtari 2600 Space InvadersScore2865.8Prior noop
Atari GamesAtari 2600 James BondScore5148Prior noop
Atari GamesAtari 2600 James BondScore3961Prior hs
Atari GamesAtari 2600 AmidarScore1838.9Prior noop
Atari GamesAtari 2600 AmidarScore129.1Prior hs
Atari GamesAtari 2600 TennisScore-5.3Prior hs
Atari GamesAtari 2600 Crazy ClimberScore141161Prior noop
Atari GamesAtari 2600 Crazy ClimberScore127512Prior hs
Atari GamesAtari 2600 AsteroidsScore2654.3Prior noop
Atari GamesAtari 2600 AsteroidsScore1745.1Prior hs
Atari GamesAtari 2600 GravitarScore548.5Prior noop
Atari GamesAtari 2600 GravitarScore269.5Prior hs
Atari GamesAtari 2600 Time PilotScore9197Prior noop
Atari GamesAtari 2600 Time PilotScore5963Prior hs
Atari GamesAtari 2600 Demon AttackScore71846.4Prior noop
Atari GamesAtari 2600 Demon AttackScore61277.5Prior hs
Atari GamesAtari 2600 Battle ZoneScore31530Prior noop
Atari GamesAtari 2600 Battle ZoneScore25520Prior hs
Atari GamesAtari 2600 Beam RiderScore31181.3Prior hs
Atari GamesAtari 2600 Beam RiderScore23384.2Prior noop
Atari GamesAtari 2600 AsterixScore31527Prior noop
Atari GamesAtari 2600 AsterixScore22484.5Prior hs
Atari GamesAtari 2600 Kung-Fu MasterScore39581Prior noop
Atari GamesAtari 2600 Kung-Fu MasterScore31676Prior hs
Atari GamesAtari 2600 BowlingScore52Prior hs
Atari GamesAtari 2600 BowlingScore47.9Prior noop
Atari GamesAtari 2600 KangarooScore16200Prior noop
Atari GamesAtari 2600 KangarooScore12185Prior hs
Atari GamesAtari 2600 AssaultScore7672.1Prior noop
Atari GamesAtari 2600 AssaultScore6548.9Prior hs
Atari GamesAtari 2600 AlienScore4203.8Prior noop
Atari GamesAtari 2600 AlienScore1334.7Prior hs
Atari GamesAtari 2600 Fishing DerbyScore39.5Prior noop
Atari GamesAtari 2600 Fishing DerbyScore9.8Prior hs
Atari GamesAtari 2600 SeaquestScore26357.8Prior noop
Atari GamesAtari 2600 SeaquestScore25463.7Prior hs
Atari GamesAtari 2600 Chopper CommandScore8600Prior noop
Atari GamesAtari 2600 Chopper CommandScore4635Prior hs
Atari GamesAtari 2600 Video PinballScore295972.8Prior hs
Atari GamesAtari 2600 Video PinballScore282007.3Prior noop
Atari GamesAtari 2600 Wizard of WorScore5727Prior hs
Atari GamesAtari 2600 Wizard of WorScore4802Prior noop
Atari GamesAtari 2600 ZaxxonScore10469Prior noop
Atari GamesAtari 2600 ZaxxonScore9474Prior hs
Atari GamesAtari 2600 RobotankScore62.6Prior noop
Atari GamesAtari 2600 RobotankScore56.2Prior hs
Atari GamesAtari 2600 Name This GameScore12270.5Prior noop
Atari GamesAtari 2600 Name This GameScore10497.6Prior hs
Atari GamesAtari 2600 Star GunnerScore63302Prior noop
Atari GamesAtari 2600 Star GunnerScore61582Prior hs
Atari GamesAtari 2600 Ice HockeyScore1.3Prior noop
Atari GamesAtari 2600 Ice HockeyScore-0.2Prior hs
Atari GamesAtari 2600 BerzerkScore1305.6Prior noop
Atari GamesAtari 2600 BerzerkScore865.9Prior hs
Atari GamesAtari 2600 AtlantisScore357324Prior noop
Atari GamesAtari 2600 AtlantisScore330647Prior hs
Atari GamesAtari 2600 HEROScore23037.7Prior noop
Atari GamesAtari 2600 HEROScore20889.9Prior hs
Atari GamesAtari 2600 Bank HeistScore1054.6Prior noop
Atari GamesAtari 2600 Bank HeistScore876.6Prior hs
Atari GamesAtari 2600 VentureScore94Prior hs
Atari GamesAtari 2600 VentureScore54Prior noop
Atari GamesAtari 2600 Private EyeScore670.7Prior hs
Atari GamesAtari 2600 Private EyeScore200Prior noop
Atari GamesAtari 2600 Q*BertScore16256.5Prior noop
Atari GamesAtari 2600 Q*BertScore9944Prior hs
Atari GamesAtari 2600 River RaidScore14522.3Prior noop
Atari GamesAtari 2600 River RaidScore11807.2Prior hs
Atari GamesAtari 2600 Road RunnerScore57608Prior noop
Atari GamesAtari 2600 Road RunnerScore52264Prior hs
Atari GamesAtari 2600 Up and DownScore16154.1Prior noop
Atari GamesAtari 2600 Up and DownScore12157.4Prior hs
Video GamesAtari 2600 BoxingScore95.6Prior noop
Video GamesAtari 2600 BoxingScore72.3Prior hs
Video GamesAtari 2600 Double DunkScore18.5Prior noop
Video GamesAtari 2600 Double DunkScore16Prior hs
Video GamesAtari 2600 Ms. PacmanScore6518.7Prior noop
Video GamesAtari 2600 Ms. PacmanScore1865.9Prior hs
Video GamesAtari 2600 CentipedeScore4463.2Prior noop
Video GamesAtari 2600 CentipedeScore3489.1Prior hs
Video GamesAtari 2600 TutankhamScore204.6Prior noop
Video GamesAtari 2600 TutankhamScore56.9Prior hs
Video GamesAtari 2600 FreewayScore33.7Prior noop
Video GamesAtari 2600 FreewayScore28.9Prior hs
Video GamesAtari 2600 PongScore20.6Prior noop
Video GamesAtari 2600 PongScore18.9Prior hs
Video GamesAtari 2600 EnduroScore2093Prior noop
Video GamesAtari 2600 EnduroScore1831Prior hs
Video GamesAtari 2600 KrullScore9728Prior noop
Video GamesAtari 2600 KrullScore6872.8Prior hs
Video GamesAtari 2600 BreakoutScore373.9Prior noop
Video GamesAtari 2600 BreakoutScore343Prior hs
Video GamesAtari 2600 FrostbiteScore4380.1Prior noop
Video GamesAtari 2600 FrostbiteScore3510Prior hs
Video GamesAtari 2600 Montezuma's RevengeScore51Prior hs
Video GamesAtari 2600 GopherScore34858.8Prior hs
Video GamesAtari 2600 GopherScore32487.2Prior noop
Video GamesAtari 2600 Space InvadersScore3912.1Prior hs
Video GamesAtari 2600 Space InvadersScore2865.8Prior noop
Video GamesAtari 2600 James BondScore5148Prior noop
Video GamesAtari 2600 James BondScore3961Prior hs
Video GamesAtari 2600 AmidarScore1838.9Prior noop
Video GamesAtari 2600 AmidarScore129.1Prior hs
Video GamesAtari 2600 TennisScore-5.3Prior hs
Video GamesAtari 2600 Crazy ClimberScore141161Prior noop
Video GamesAtari 2600 Crazy ClimberScore127512Prior hs
Video GamesAtari 2600 AsteroidsScore2654.3Prior noop
Video GamesAtari 2600 AsteroidsScore1745.1Prior hs
Video GamesAtari 2600 GravitarScore548.5Prior noop
Video GamesAtari 2600 GravitarScore269.5Prior hs
Video GamesAtari 2600 Time PilotScore9197Prior noop
Video GamesAtari 2600 Time PilotScore5963Prior hs
Video GamesAtari 2600 Demon AttackScore71846.4Prior noop
Video GamesAtari 2600 Demon AttackScore61277.5Prior hs
Video GamesAtari 2600 Battle ZoneScore31530Prior noop
Video GamesAtari 2600 Battle ZoneScore25520Prior hs
Video GamesAtari 2600 Beam RiderScore31181.3Prior hs
Video GamesAtari 2600 Beam RiderScore23384.2Prior noop
Video GamesAtari 2600 AsterixScore31527Prior noop
Video GamesAtari 2600 AsterixScore22484.5Prior hs
Video GamesAtari 2600 Kung-Fu MasterScore39581Prior noop
Video GamesAtari 2600 Kung-Fu MasterScore31676Prior hs
Video GamesAtari 2600 BowlingScore52Prior hs
Video GamesAtari 2600 BowlingScore47.9Prior noop
Video GamesAtari 2600 KangarooScore16200Prior noop
Video GamesAtari 2600 KangarooScore12185Prior hs
Video GamesAtari 2600 AssaultScore7672.1Prior noop
Video GamesAtari 2600 AssaultScore6548.9Prior hs
Video GamesAtari 2600 AlienScore4203.8Prior noop
Video GamesAtari 2600 AlienScore1334.7Prior hs
Video GamesAtari 2600 Fishing DerbyScore39.5Prior noop
Video GamesAtari 2600 Fishing DerbyScore9.8Prior hs
Video GamesAtari 2600 SeaquestScore26357.8Prior noop
Video GamesAtari 2600 SeaquestScore25463.7Prior hs
Video GamesAtari 2600 Chopper CommandScore8600Prior noop
Video GamesAtari 2600 Chopper CommandScore4635Prior hs
Video GamesAtari 2600 Video PinballScore295972.8Prior hs
Video GamesAtari 2600 Video PinballScore282007.3Prior noop
Video GamesAtari 2600 Wizard of WorScore5727Prior hs
Video GamesAtari 2600 Wizard of WorScore4802Prior noop
Video GamesAtari 2600 ZaxxonScore10469Prior noop
Video GamesAtari 2600 ZaxxonScore9474Prior hs
Video GamesAtari 2600 RobotankScore62.6Prior noop
Video GamesAtari 2600 RobotankScore56.2Prior hs
Video GamesAtari 2600 Name This GameScore12270.5Prior noop
Video GamesAtari 2600 Name This GameScore10497.6Prior hs
Video GamesAtari 2600 Star GunnerScore63302Prior noop
Video GamesAtari 2600 Star GunnerScore61582Prior hs
Video GamesAtari 2600 Ice HockeyScore1.3Prior noop
Video GamesAtari 2600 Ice HockeyScore-0.2Prior hs
Video GamesAtari 2600 BerzerkScore1305.6Prior noop
Video GamesAtari 2600 BerzerkScore865.9Prior hs
Video GamesAtari 2600 AtlantisScore357324Prior noop
Video GamesAtari 2600 AtlantisScore330647Prior hs
Video GamesAtari 2600 HEROScore23037.7Prior noop
Video GamesAtari 2600 HEROScore20889.9Prior hs
Video GamesAtari 2600 Bank HeistScore1054.6Prior noop
Video GamesAtari 2600 Bank HeistScore876.6Prior hs
Video GamesAtari 2600 VentureScore94Prior hs
Video GamesAtari 2600 VentureScore54Prior noop
Video GamesAtari 2600 Private EyeScore670.7Prior hs
Video GamesAtari 2600 Private EyeScore200Prior noop
Video GamesAtari 2600 Q*BertScore16256.5Prior noop
Video GamesAtari 2600 Q*BertScore9944Prior hs
Video GamesAtari 2600 River RaidScore14522.3Prior noop
Video GamesAtari 2600 River RaidScore11807.2Prior hs
Video GamesAtari 2600 Road RunnerScore57608Prior noop
Video GamesAtari 2600 Road RunnerScore52264Prior hs
Video GamesAtari 2600 Up and DownScore16154.1Prior noop
Video GamesAtari 2600 Up and DownScore12157.4Prior hs

Related Papers

CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning2025-07-18VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning2025-07-17Spectral Bellman Method: Unifying Representation and Exploration in RL2025-07-17Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback2025-07-17VAR-MATH: Probing True Mathematical Reasoning in Large Language Models via Symbolic Multi-Instance Benchmarks2025-07-17QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation2025-07-17Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities2025-07-17Autonomous Resource Management in Microservice Systems via Reinforcement Learning2025-07-17