Deep Exploration via Bootstrapped DQN

Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van Roy

2016-02-15NeurIPS 2016 12Reinforcement Learning Atari Games Efficient Exploration reinforcement-learning

Abstract

Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.

Results

Task	Dataset	Metric	Value	Model
Atari Games	Atari 2600 Boxing	Score	93.2	Bootstrapped DQN
Atari Games	Atari 2600 Double Dunk	Score	3	Bootstrapped DQN
Atari Games	Atari 2600 Ms. Pacman	Score	2983.3	Bootstrapped DQN
Atari Games	Atari 2600 Centipede	Score	4553.5	Bootstrapped DQN
Atari Games	Atari 2600 Tutankham	Score	214.8	Bootstrapped DQN
Atari Games	Atari 2600 Freeway	Score	33.9	Bootstrapped DQN
Atari Games	Atari 2600 Pong	Score	20.9	Bootstrapped DQN
Atari Games	Atari 2600 Enduro	Score	1591	Bootstrapped DQN
Atari Games	Atari 2600 Krull	Score	8627.9	Bootstrapped DQN
Atari Games	Atari 2600 Breakout	Score	855	Bootstrapped DQN
Atari Games	Atari 2600 Frostbite	Score	2181.4	Bootstrapped DQN
Atari Games	Atari 2600 Montezuma's Revenge	Score	100	Bootstrapped DQN
Atari Games	Atari 2600 Gopher	Score	17438.4	Bootstrapped DQN
Atari Games	Atari 2600 Space Invaders	Score	2893	Bootstrapped DQN
Atari Games	Atari 2600 James Bond	Score	1663.5	Bootstrapped DQN
Atari Games	Atari 2600 Amidar	Score	1272.5	Bootstrapped DQN
Atari Games	Atari 2600 Crazy Climber	Score	137925.9	Bootstrapped DQN
Atari Games	Atari 2600 Asteroids	Score	1032	Bootstrapped DQN
Atari Games	Atari 2600 Gravitar	Score	286.1	Bootstrapped DQN
Atari Games	Atari 2600 Time Pilot	Score	9079.4	Bootstrapped DQN
Atari Games	Atari 2600 Demon Attack	Score	82610	Bootstrapped DQN
Atari Games	Atari 2600 Battle Zone	Score	38666.7	Bootstrapped DQN
Atari Games	Atari 2600 Beam Rider	Score	23429.8	Bootstrapped DQN
Atari Games	Atari 2600 Asterix	Score	19713.2	Bootstrapped DQN
Atari Games	Atari 2600 Kung-Fu Master	Score	36733.3	Bootstrapped DQN
Atari Games	Atari 2600 Bowling	Score	60.2	Bootstrapped DQN
Atari Games	Atari 2600 Kangaroo	Score	14862.5	Bootstrapped DQN
Atari Games	Atari 2600 Assault	Score	8047.1	Bootstrapped DQN
Atari Games	Atari 2600 Alien	Score	2436.6	Bootstrapped DQN
Atari Games	Atari 2600 Fishing Derby	Score	26	Bootstrapped DQN
Atari Games	Atari 2600 Seaquest	Score	9083.1	Bootstrapped DQN
Atari Games	Atari 2600 Chopper Command	Score	4100	Bootstrapped DQN
Atari Games	Atari 2600 Video Pinball	Score	811610	Bootstrapped DQN
Atari Games	Atari 2600 Wizard of Wor	Score	6804.7	Bootstrapped DQN
Atari Games	Atari 2600 Zaxxon	Score	11491.7	Bootstrapped DQN
Atari Games	Atari 2600 Robotank	Score	66.6	Bootstrapped DQN
Atari Games	Atari 2600 Name This Game	Score	11501.1	Bootstrapped DQN
Atari Games	Atari 2600 Star Gunner	Score	55725	Bootstrapped DQN
Atari Games	Atari 2600 Ice Hockey	Score	-1.3	Bootstrapped DQN
Atari Games	Atari 2600 Atlantis	Score	994500	Bootstrapped DQN
Atari Games	Atari 2600 HERO	Score	21021.3	Bootstrapped DQN
Atari Games	Atari 2600 Bank Heist	Score	1208	Bootstrapped DQN
Atari Games	Atari 2600 Venture	Score	212.5	Bootstrapped DQN
Atari Games	Atari 2600 Private Eye	Score	1812.5	Bootstrapped DQN
Atari Games	Atari 2600 Q*Bert	Score	15092.7	Bootstrapped DQN
Atari Games	Atari 2600 River Raid	Score	12845	Bootstrapped DQN
Atari Games	Atari 2600 Road Runner	Score	51500	Bootstrapped DQN
Atari Games	Atari 2600 Up and Down	Score	26231	Bootstrapped DQN
Video Games	Atari 2600 Boxing	Score	93.2	Bootstrapped DQN
Video Games	Atari 2600 Double Dunk	Score	3	Bootstrapped DQN
Video Games	Atari 2600 Ms. Pacman	Score	2983.3	Bootstrapped DQN
Video Games	Atari 2600 Centipede	Score	4553.5	Bootstrapped DQN
Video Games	Atari 2600 Tutankham	Score	214.8	Bootstrapped DQN
Video Games	Atari 2600 Freeway	Score	33.9	Bootstrapped DQN
Video Games	Atari 2600 Pong	Score	20.9	Bootstrapped DQN
Video Games	Atari 2600 Enduro	Score	1591	Bootstrapped DQN
Video Games	Atari 2600 Krull	Score	8627.9	Bootstrapped DQN
Video Games	Atari 2600 Breakout	Score	855	Bootstrapped DQN
Video Games	Atari 2600 Frostbite	Score	2181.4	Bootstrapped DQN
Video Games	Atari 2600 Montezuma's Revenge	Score	100	Bootstrapped DQN
Video Games	Atari 2600 Gopher	Score	17438.4	Bootstrapped DQN
Video Games	Atari 2600 Space Invaders	Score	2893	Bootstrapped DQN
Video Games	Atari 2600 James Bond	Score	1663.5	Bootstrapped DQN
Video Games	Atari 2600 Amidar	Score	1272.5	Bootstrapped DQN
Video Games	Atari 2600 Crazy Climber	Score	137925.9	Bootstrapped DQN
Video Games	Atari 2600 Asteroids	Score	1032	Bootstrapped DQN
Video Games	Atari 2600 Gravitar	Score	286.1	Bootstrapped DQN
Video Games	Atari 2600 Time Pilot	Score	9079.4	Bootstrapped DQN
Video Games	Atari 2600 Demon Attack	Score	82610	Bootstrapped DQN
Video Games	Atari 2600 Battle Zone	Score	38666.7	Bootstrapped DQN
Video Games	Atari 2600 Beam Rider	Score	23429.8	Bootstrapped DQN
Video Games	Atari 2600 Asterix	Score	19713.2	Bootstrapped DQN
Video Games	Atari 2600 Kung-Fu Master	Score	36733.3	Bootstrapped DQN
Video Games	Atari 2600 Bowling	Score	60.2	Bootstrapped DQN
Video Games	Atari 2600 Kangaroo	Score	14862.5	Bootstrapped DQN
Video Games	Atari 2600 Assault	Score	8047.1	Bootstrapped DQN
Video Games	Atari 2600 Alien	Score	2436.6	Bootstrapped DQN
Video Games	Atari 2600 Fishing Derby	Score	26	Bootstrapped DQN
Video Games	Atari 2600 Seaquest	Score	9083.1	Bootstrapped DQN
Video Games	Atari 2600 Chopper Command	Score	4100	Bootstrapped DQN
Video Games	Atari 2600 Video Pinball	Score	811610	Bootstrapped DQN
Video Games	Atari 2600 Wizard of Wor	Score	6804.7	Bootstrapped DQN
Video Games	Atari 2600 Zaxxon	Score	11491.7	Bootstrapped DQN
Video Games	Atari 2600 Robotank	Score	66.6	Bootstrapped DQN
Video Games	Atari 2600 Name This Game	Score	11501.1	Bootstrapped DQN
Video Games	Atari 2600 Star Gunner	Score	55725	Bootstrapped DQN
Video Games	Atari 2600 Ice Hockey	Score	-1.3	Bootstrapped DQN
Video Games	Atari 2600 Atlantis	Score	994500	Bootstrapped DQN
Video Games	Atari 2600 HERO	Score	21021.3	Bootstrapped DQN
Video Games	Atari 2600 Bank Heist	Score	1208	Bootstrapped DQN
Video Games	Atari 2600 Venture	Score	212.5	Bootstrapped DQN
Video Games	Atari 2600 Private Eye	Score	1812.5	Bootstrapped DQN
Video Games	Atari 2600 Q*Bert	Score	15092.7	Bootstrapped DQN
Video Games	Atari 2600 River Raid	Score	12845	Bootstrapped DQN
Video Games	Atari 2600 Road Runner	Score	51500	Bootstrapped DQN
Video Games	Atari 2600 Up and Down	Score	26231	Bootstrapped DQN

Deep Exploration via Bootstrapped DQN

Abstract

Results

Related Papers

Deep Exploration via Bootstrapped DQN

Abstract

Results

Related Papers