RMSProp

RMSProp is an unpublished adaptive learning rate optimizer proposed by Geoff Hinton. The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:

$E\left[g^{2}\right]\_{t} = \gamma E\left[g^{2}\right]\_{t-1} + \left(1 - \gamma\right) g^{2}\_{t}$

$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]\_{t} + \epsilon}}g\_{t}$

Hinton suggests $\gamma=0.9$, with a good default for $\eta$ as $0.001$.

Image: Alec Radford