Language Modelling on LAMBADA

Metric: Accuracy (higher is better)

LeaderboardDataset

Loading chart...

Results

Sort:

#	Model↕	Accuracy▼	Extra Data	Paper	Date↕	Code
1	PaLM-540B (Few-Shot)	89.7	No	PaLM: Scaling Language Modeling with Pathways	2022-04-05	Code
2	PaLM 2-L (one-shot)	86.9	No	PaLM 2 Technical Report	2023-05-17	Code
3	GPT-3 175B (Few-Shot)	86.4	No	Language Models are Few-Shot Learners	2020-05-28	Code
4	LLaMA-65B+CFG (Zero-Shot)	84	No	Stay on topic with Classifier-Free Guidance	2023-06-30	-
5	LLaMA-30B+CFG (zero-shot)	83.9	No	Stay on topic with Classifier-Free Guidance	2023-06-30	-
6	PaLM 2-M (one-shot)	83.7	No	PaLM 2 Technical Report	2023-05-17	Code
7	Cohere Large	82.33	No	-	-	-
8	LLaMA-13B+CFG (zero-shot)	82.2	No	Stay on topic with Classifier-Free Guidance	2023-06-30	-
9	PaLM-540B (One-Shot)	81.8	No	PaLM: Scaling Language Modeling with Pathways	2022-04-05	Code
10	GLaM 62B/64E (One-Shot)	80.9	No	GLaM: Efficient Scaling of Language Models with ...	2021-12-13	-
11	PaLM 2-S (one-shot)	80.7	No	PaLM 2 Technical Report	2023-05-17	Code
12	GLM-130B (bidirectional attention)	80.2	No	GLM-130B: An Open Bilingual Pre-trained Model	2022-10-05	Code
13	SparseGPT (175B, 2:4 Sparsity)	79.47	No	SparseGPT: Massive Language Models Can Be Accura...	2023-01-02	Code
14	SparseGPT (175B, 4:8 Sparsity)	78.77	No	SparseGPT: Massive Language Models Can Be Accura...	2023-01-02	Code
15	PaLM-540B (Zero-Shot)	77.9	No	PaLM: Scaling Language Modeling with Pathways	2022-04-05	Code
16	Chinchilla (Zero-Shot)	77.7	No	Training Compute-Optimal Large Language Models	2022-03-29	Code
17	SparseGPT (175B, 50% Sparsity)	76.51	No	SparseGPT: Massive Language Models Can Be Accura...	2023-01-02	Code
18	GPT-3 175B (Zero-Shot)	76.2	No	Language Models are Few-Shot Learners	2020-05-28	Code
19	OPT-175B	75.59	No	SparseGPT: Massive Language Models Can Be Accura...	2023-01-02	Code
20	GPT-3 13B (Zero-Shot)	72.5	No	Language Models are Few-Shot Learners	2020-05-28	Code
21	GLM-XXLarge (bidirectional)	72.35	No	GLM: General Language Model Pretraining with Aut...	2021-03-18	Code
22	Pythia 12B (0-shot)	70.46	No	Pythia: A Suite for Analyzing Large Language Mod...	2023-04-03	Code
23	GPT-3 6.7B (Zero-Shot)	70.3	No	Language Models are Few-Shot Learners	2020-05-28	Code
24	GPT-J-6B	69.7	No	-	-	-
25	Mamba-2.8B	69.2	No	Mamba: Linear-Time Sequence Modeling with Select...	2023-12-01	Code
26	Pythia 6.9B (0-shot)	67.28	No	Pythia: A Suite for Analyzing Large Language Mod...	2023-04-03	Code
27	GLM-XXLarge (unidirectional)	67.18	No	GLM: General Language Model Pretraining with Aut...	2021-03-18	Code
28	GPT-3 2.7B (Zero-Shot)	67.1	No	Language Models are Few-Shot Learners	2020-05-28	Code
29	GPT-2 1.5B (Zero Shot)	63.24	No	-	-	Code
30	Universal Transformer (w/ dynamic halting)	56.25	No	Universal Transformers	2018-07-10	Code
31	Residual Shuffle-Exchange network	54.34	No	Residual Shuffle-Exchange Networks for Fast Proc...	2020-04-06	Code
32	Gated-Attention Reader (+ features)	49	No	Broad Context Language Modeling as Reading Compr...	2016-10-26	-
33	OPT-175B (50% Sparsity)	0.02	No	SparseGPT: Massive Language Models Can Be Accura...	2023-01-02	Code
34	test	0.01	No	Test-Time Training with Self-Supervision for Gen...	2019-09-29	Code

#1PaLM-540B (Few-Shot)SOTA
89.7
Accuracy· 2022-04-05
PaLM: Scaling Language Modeling with Pathways Code
#2PaLM 2-L (one-shot)
86.9
Accuracy· 2023-05-17
PaLM 2 Technical Report Code
#3GPT-3 175B (Few-Shot)SOTA
86.4
Accuracy· 2020-05-28
Language Models are Few-Shot Learners Code
#4LLaMA-65B+CFG (Zero-Shot)
84
Accuracy· 2023-06-30
Stay on topic with Classifier-Free Guidance
#5LLaMA-30B+CFG (zero-shot)
83.9
Accuracy· 2023-06-30
Stay on topic with Classifier-Free Guidance
#6PaLM 2-M (one-shot)
83.7
Accuracy· 2023-05-17
PaLM 2 Technical Report Code
#7Cohere Large
82.33
Accuracy
No paper
#8LLaMA-13B+CFG (zero-shot)
82.2
Accuracy· 2023-06-30
Stay on topic with Classifier-Free Guidance
#9PaLM-540B (One-Shot)
81.8
Accuracy· 2022-04-05
PaLM: Scaling Language Modeling with Pathways Code
#10GLaM 62B/64E (One-Shot)
80.9
Accuracy· 2021-12-13
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
#11PaLM 2-S (one-shot)
80.7
Accuracy· 2023-05-17
PaLM 2 Technical Report Code
#12GLM-130B (bidirectional attention)
80.2
Accuracy· 2022-10-05
GLM-130B: An Open Bilingual Pre-trained Model Code
#13SparseGPT (175B, 2:4 Sparsity)
79.47
Accuracy· 2023-01-02
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Code
#14SparseGPT (175B, 4:8 Sparsity)
78.77
Accuracy· 2023-01-02
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Code
#15PaLM-540B (Zero-Shot)
77.9
Accuracy· 2022-04-05
PaLM: Scaling Language Modeling with Pathways Code
#16Chinchilla (Zero-Shot)
77.7
Accuracy· 2022-03-29
Training Compute-Optimal Large Language Models Code
#17SparseGPT (175B, 50% Sparsity)
76.51
Accuracy· 2023-01-02
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Code
#18GPT-3 175B (Zero-Shot)
76.2
Accuracy· 2020-05-28
Language Models are Few-Shot Learners Code
#19OPT-175B
75.59
Accuracy· 2023-01-02
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Code
#20GPT-3 13B (Zero-Shot)
72.5
Accuracy· 2020-05-28
Language Models are Few-Shot Learners Code
#21GLM-XXLarge (bidirectional)
72.35
Accuracy· 2021-03-18
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Code
#22Pythia 12B (0-shot)
70.46
Accuracy· 2023-04-03
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Code
#23GPT-3 6.7B (Zero-Shot)
70.3
Accuracy· 2020-05-28
Language Models are Few-Shot Learners Code
#24GPT-J-6B
69.7
Accuracy
No paper
#25Mamba-2.8B
69.2
Accuracy· 2023-12-01
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Code
#26Pythia 6.9B (0-shot)
67.28
Accuracy· 2023-04-03
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Code
#27GLM-XXLarge (unidirectional)
67.18
Accuracy· 2021-03-18
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Code
#28GPT-3 2.7B (Zero-Shot)
67.1
Accuracy· 2020-05-28
Language Models are Few-Shot Learners Code
#29GPT-2 1.5B (Zero Shot)
63.24
Accuracy
No paperCode
#30Universal Transformer (w/ dynamic halting)SOTA
56.25
Accuracy· 2018-07-10
Universal Transformers Code
#31Residual Shuffle-Exchange network
54.34
Accuracy· 2020-04-06
Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences Code
#32Gated-Attention Reader (+ features)SOTA
49
Accuracy· 2016-10-26
Broad Context Language Modeling as Reading Comprehension
#33OPT-175B (50% Sparsity)
0.02
Accuracy· 2023-01-02
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Code
#34test
0.01
Accuracy· 2019-09-29
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts Code