AggMo

Aggregated Momentum (AggMo) is a variant of the classical momentum stochastic optimizer which maintains several velocity vectors with different $\beta$ parameters. AggMo averages the velocity vectors when updating the parameters. It resolves the problem of choosing a momentum parameter by taking a linear combination of multiple momentum buffers. Each of $K$ momentum buffers have a different discount factor $\beta \in \mathbb{R}^{K}$, and these are averaged for the update. The update rule is:

$\textbf{v}\_{t}^{\left(i\right)} = \beta^{(i)}\textbf{v}\_{t-1}^{\left(i\right)} - \nabla\_{\theta}f\left(\mathbf{\theta}\_{t-1}\right)$

$\mathbf{\theta\_{t}} = \mathbf{\theta\_{t-1}} + \frac{\gamma\_{t}}{K}\sum^{K}\_{i=1}\textbf{v}\_{t}^{\left(i\right)}$

where $v^{\left(i\right)}_{0}$ for each $i$. The vector $\mathcal{\beta} = \left[\beta^{(1)}, \ldots, \beta^{(K)}\right]$ is the dampening factor.