∞-former (Sticky memories + initialized GPT-2 Small)

Reported on 5 benchmarks across 5 tasks · 1 paper · 4 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Speech1 result

DialogueonPG-19
Perplexity· uses extra data· 2021-09-01
32.48
SOTA
$\infty$-former: Infinite Memory Transformer arXiv:2109.00301

Adversarial1 result

Text GenerationonPG-19
Perplexity· uses extra data· 2021-09-01
32.48
SOTA
$\infty$-former: Infinite Memory Transformer arXiv:2109.00301

Methodology1 result

ChatbotonPG-19
Perplexity· uses extra data· 2021-09-01
32.48
SOTA
$\infty$-former: Infinite Memory Transformer arXiv:2109.00301

Natural Language Processing1 result

Dialogue GenerationonPG-19
Perplexity· uses extra data· 2021-09-01
32.48
SOTA
$\infty$-former: Infinite Memory Transformer arXiv:2109.00301

Medical1 result

Language ModellingonWikiText-103
Test perplexity· uses extra data· 2021-09-01
16.61
best: 2.4 (RETRO (7.5B))
$\infty$-former: Infinite Memory Transformer arXiv:2109.00301