Double Q-learning

Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning.

The max operator in standard Q-learning and DQN uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation, which is the idea behind Double Q-learning:

$Y^{Q}\_{t} = R\_{t+1} + \gamma{Q}\left(S\_{t+1}, \arg\max\_{a}Q\left(S\_{t+1}, a; \mathbb{\theta}\_{t}\right);\mathbb{\theta}\_{t}\right)$

The Double Q-learning error can then be written as:

$Y^{DoubleQ}\_{t} = R\_{t+1} + \gamma{Q}\left(S\_{t+1}, \arg\max\_{a}Q\left(S\_{t+1}, a; \mathbb{\theta}\_{t}\right);\mathbb{\theta}^{'}\_{t}\right)$

Here the selection of the action in the $\arg\max$ is still due to the online weights $\theta\_{t}$. But we use a second set of weights $\mathbb{\theta}^{'}\_{t}$ to fairly evaluate the value of this policy.

Source: Deep Reinforcement Learning with Double Q-learning