BEIR

BEIR (Benchmarking IR) is a heterogeneous benchmark containing different information retrieval (IR) tasks. Through BEIR, it is possible to systematically study the zero-shot generalization capabilities of multiple neural retrieval approaches.

The benchmark contains a total of 9 information retrieval tasks (Fact Checking, Citation Prediction, Duplicate Question Retrieval, Argument Retrieval, News Retrieval, Question Answering, Tweet Retrieval, Biomedical IR, Entity Retrieval) from 19 different datasets:

• MS MARCO
• TREC-COVID
• NFCorpus
• BioASQ
• Natural Questions
• HotpotQA
• FiQA-2018
• Signal-1M
• TREC-News
• ArguAna
• Touche 2020
• CQADupStack
• Quora Question Pairs
• DBPedia
• SciDocs
• FEVER
• Climate-FEVER
• SciFact
• Robust04