Polyak Averaging

Polyak Averaging is an optimization technique that sets final parameters to an average of (recent) parameters visited in the optimization trajectory. Specifically if in $t$ iterations we have parameters $\theta\_{1}, \theta\_{2}, \dots, \theta\_{t}$, then Polyak Averaging suggests setting

$\theta\_t =\frac{1}{t}\sum\_{i}\theta\_{i}$

Image Credit: Shubhendu Trivedi & Risi Kondor