Question Answering on SQuAD1.1 dev

Metric: F1 (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	F1▼	Extra Data	Paper	Date↕	Code
1	XLNet+DSC	95.77	Yes	Dice Loss for Data-imbalanced NLP Tasks	2019-11-07	Code
2	T5-11B	95.64	Yes	Exploring the Limits of Transfer Learning with a...	2019-10-23	Code
3	XLNet (single model)	95.1	Yes	XLNet: Generalized Autoregressive Pretraining fo...	2019-06-19	Code
4	LUKE 483M	95	No	LUKE: Deep Contextualized Entity Representations...	2020-10-02	Code
5	T5-3B	94.95	Yes	Exploring the Limits of Transfer Learning with a...	2019-10-23	Code
6	T5-Large 770M	93.79	No	Exploring the Limits of Transfer Learning with a...	2019-10-23	Code
7	BERT-LARGE (Ensemble+TriviaQA)	92.2	No	BERT: Pre-training of Deep Bidirectional Transfo...	2018-10-11	Code
8	T5-Base	92.08	Yes	Exploring the Limits of Transfer Learning with a...	2019-10-23	Code
9	BERT-LARGE (Single+TriviaQA)	91.1	No	BERT: Pre-training of Deep Bidirectional Transfo...	2018-10-11	Code
10	BART Base (with text infilling)	90.8	No	BART: Denoising Sequence-to-Sequence Pre-trainin...	2019-10-29	Code
11	BERT large (LAMB optimizer)	90.584	No	Large Batch Optimization for Deep Learning: Trai...	2019-04-01	Code
12	BERT-Large-uncased-PruneOFA (90% unstruct sparse)	90.2	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
13	BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)	90.02	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
14	BERT-Base-uncased-PruneOFA (85% unstruct sparse)	88.42	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
15	BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)	88.24	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
16	TinyBERT-6 67M	87.5	No	TinyBERT: Distilling BERT for Natural Language U...	2019-09-23	Code
17	BERT-Base-uncased-PruneOFA (90% unstruct sparse)	87.25	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
18	T5-Small	87.24	Yes	Exploring the Limits of Transfer Learning with a...	2019-10-23	Code
19	R.M-Reader (single)	86.3	No	Reinforced Mnemonic Reader for Machine Reading C...	2017-05-08	Code
20	DensePhrases	86.3	No	Learning Dense Representations of Phrases at Scale	2020-12-23	Code
21	DistilBERT-uncased-PruneOFA (85% unstruct sparse)	85.82	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
22	DistilBERT 66M	85.8	No	DistilBERT, a distilled version of BERT: smaller...	2019-10-02	Code
23	BiDAF + Self Attention + ELMo	85.6	No	Deep contextualized word representations	2018-02-15	Code
24	DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)	85.13	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
25	KAR	84.9	No	Explicit Utilization of General Knowledge in Mac...	2018-09-10	-
26	DistilBERT-uncased-PruneOFA (90% unstruct sparse)	84.82	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
27	SAN (single)	84.056	No	Stochastic Answer Networks for Machine Reading C...	2017-12-10	Code
28	DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)	83.87	No	Prune Once for All: Sparse Pre-Trained Language ...	2021-11-10	Code
29	QANet (data aug x3)	83.8	No	QANet: Combining Local Convolution with Global S...	2018-04-23	Code
30	FusionNet	83.6	No	FusionNet: Fusing via Fully-Aware Attention with...	2017-11-16	Code
31	QANet (data aug x2)	83.2	No	QANet: Combining Local Convolution with Global S...	2018-04-23	Code
32	DCN+ (single)	83.1	No	DCN+: Mixed Objective and Deep Residual Coattent...	2017-10-31	Code
33	QANet	82.7	No	QANet: Combining Local Convolution with Global S...	2018-04-23	Code
34	PhaseCond (single)	81.4	No	Phase Conductor on Multi-layered Attentions for ...	2017-10-28	-
35	SRU	80.2	No	Simple Recurrent Units for Highly Parallelizable...	2017-09-08	Code
36	Smarnet	80.183	No	Smarnet: Teaching Machines to Read and Comprehen...	2017-10-08	-
37	DCN (Char + CoVe)	79.9	No	Learned in Translation: Contextualized Word Vect...	2017-08-01	Code
38	R-NET (single)	79.5	No	-	-	-
39	Ruminating Reader	79.5	No	Ruminating Reader: Reasoning with Gated Multi-Ho...	2017-04-24	-
40	DrQA (Document Reader only)	78.8	No	Reading Wikipedia to Answer Open-Domain Questions	2017-03-31	Code
41	FastQAExt (beam-size 5)	78.5	No	Making Neural QA as Simple as Possible but not S...	2017-03-14	Code
42	jNet (TreeLSTM adaptation, QTLa, K=100)	78.38	No	Exploring Question Understanding and Adaptation ...	2017-03-14	-
43	SEDT-LSTM	77.42	No	Structural Embedding of Syntactic Trees for Mach...	2017-03-02	-
44	BIDAF (single)	77.3	No	Bidirectional Attention Flow for Machine Compreh...	2016-11-05	Code
45	SECT-LSTM	77.19	No	Structural Embedding of Syntactic Trees for Mach...	2017-03-02	-
46	MPCM	75.8	No	Multi-Perspective Context Matching for Machine C...	2016-12-13	Code
47	DCN	75.6	No	Dynamic Coattention Networks For Question Answer...	2016-11-05	Code
48	FABIR	75.6	No	A Fully Attention-Based Information Retriever	2018-10-22	Code
49	RASOR	74.9	No	Learning Recurrent Span Representations for Extr...	2016-11-04	Code
50	FG fine-grained gate	71.25	No	Words or Characters? Fine-grained Gating for Rea...	2016-11-06	Code
51	DCR	71.2	No	End-to-End Answer Chunk Extraction and Ranking f...	2016-10-31	-
52	Match-LSTM with Bi-Ans-Ptr (Boundary+Search+b)	64.7	No	Machine Comprehension Using Match-LSTM and Answe...	2016-08-29	Code

#1XLNet+DSCSOTA
95.77
F1· Extra Data· 2019-11-07
Dice Loss for Data-imbalanced NLP Tasks Code
#2T5-11BSOTA
95.64
F1· Extra Data· 2019-10-23
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Code
#3XLNet (single model)SOTA
95.1
F1· Extra Data· 2019-06-19
XLNet: Generalized Autoregressive Pretraining for Language Understanding Code
#4LUKE 483M
95
F1· 2020-10-02
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention Code
#5T5-3B
94.95
F1· Extra Data· 2019-10-23
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Code
#6T5-Large 770M
93.79
F1· 2019-10-23
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Code
#7BERT-LARGE (Ensemble+TriviaQA)SOTA
92.2
F1· 2018-10-11
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Code
#8T5-Base
92.08
F1· Extra Data· 2019-10-23
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Code
#9BERT-LARGE (Single+TriviaQA)
91.1
F1· 2018-10-11
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Code
#10BART Base (with text infilling)
90.8
F1· 2019-10-29
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension Code
#11BERT large (LAMB optimizer)
90.584
F1· 2019-04-01
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Code
#12BERT-Large-uncased-PruneOFA (90% unstruct sparse)
90.2
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#13BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)
90.02
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#14BERT-Base-uncased-PruneOFA (85% unstruct sparse)
88.42
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#15BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)
88.24
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#16TinyBERT-6 67M
87.5
F1· 2019-09-23
TinyBERT: Distilling BERT for Natural Language Understanding Code
#17BERT-Base-uncased-PruneOFA (90% unstruct sparse)
87.25
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#18T5-Small
87.24
F1· Extra Data· 2019-10-23
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Code
#19R.M-Reader (single)SOTA
86.3
F1· 2017-05-08
Reinforced Mnemonic Reader for Machine Reading Comprehension Code
#20DensePhrases
86.3
F1· 2020-12-23
Learning Dense Representations of Phrases at Scale Code
#21DistilBERT-uncased-PruneOFA (85% unstruct sparse)
85.82
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#22DistilBERT 66M
85.8
F1· 2019-10-02
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Code
#23BiDAF + Self Attention + ELMo
85.6
F1· 2018-02-15
Deep contextualized word representations Code
#24DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)
85.13
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#25KAR
84.9
F1· 2018-09-10
Explicit Utilization of General Knowledge in Machine Reading Comprehension
#26DistilBERT-uncased-PruneOFA (90% unstruct sparse)
84.82
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#27SAN (single)
84.056
F1· 2017-12-10
Stochastic Answer Networks for Machine Reading Comprehension Code
#28DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)
83.87
F1· 2021-11-10
Prune Once for All: Sparse Pre-Trained Language Models Code
#29QANet (data aug x3)
83.8
F1· 2018-04-23
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension Code
#30FusionNet
83.6
F1· 2017-11-16
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension Code
#31QANet (data aug x2)
83.2
F1· 2018-04-23
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension Code
#32DCN+ (single)
83.1
F1· 2017-10-31
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering Code
#33QANet
82.7
F1· 2018-04-23
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension Code
#34PhaseCond (single)
81.4
F1· 2017-10-28
Phase Conductor on Multi-layered Attentions for Machine Comprehension
#35SRU
80.2
F1· 2017-09-08
Simple Recurrent Units for Highly Parallelizable Recurrence Code
#36Smarnet
80.183
F1· 2017-10-08
Smarnet: Teaching Machines to Read and Comprehend Like Human
#37DCN (Char + CoVe)
79.9
F1· 2017-08-01
Learned in Translation: Contextualized Word Vectors Code
#38R-NET (single)
79.5
F1
No paper
#39Ruminating ReaderSOTA
79.5
F1· 2017-04-24
Ruminating Reader: Reasoning with Gated Multi-Hop Attention
#40DrQA (Document Reader only)SOTA
78.8
F1· 2017-03-31
Reading Wikipedia to Answer Open-Domain Questions Code
#41FastQAExt (beam-size 5)SOTA
78.5
F1· 2017-03-14
Making Neural QA as Simple as Possible but not Simpler Code
#42jNet (TreeLSTM adaptation, QTLa, K=100)
78.38
F1· 2017-03-14
Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
#43SEDT-LSTMSOTA
77.42
F1· 2017-03-02
Structural Embedding of Syntactic Trees for Machine Comprehension
#44BIDAF (single)SOTA
77.3
F1· 2016-11-05
Bidirectional Attention Flow for Machine Comprehension Code
#45SECT-LSTM
77.19
F1· 2017-03-02
Structural Embedding of Syntactic Trees for Machine Comprehension
#46MPCM
75.8
F1· 2016-12-13
Multi-Perspective Context Matching for Machine Comprehension Code
#47DCN
75.6
F1· 2016-11-05
Dynamic Coattention Networks For Question Answering Code
#48FABIR
75.6
F1· 2018-10-22
A Fully Attention-Based Information Retriever Code
#49RASORSOTA
74.9
F1· 2016-11-04
Learning Recurrent Span Representations for Extractive Question Answering Code
#50FG fine-grained gate
71.25
F1· 2016-11-06
Words or Characters? Fine-grained Gating for Reading Comprehension Code
#51DCRSOTA
71.2
F1· 2016-10-31
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
#52Match-LSTM with Bi-Ans-Ptr (Boundary+Search+b) SOTA
64.7
F1· 2016-08-29
Machine Comprehension Using Match-LSTM and Answer Pointer Code