Tasks SotA Datasets Papers Methods Submit About

Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable Benchmarks All SotA Datasets Papers Methods

Community

Submit Results About

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Datasets/BIG-bench

BIG-bench

Beyond the Imitation Game Benchmark

TextsApache License 2.0Introduced 2022-06-09

The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities. Big-bench include more than 200 tasks.

Image source: https://arxiv.org/pdf/2206.04615.pdf

Benchmarks

Analogical Similarity/Accuracy Anatomy/Accuracy Astronomy/Accuracy BIG-bench Machine Learning/Accuracy Clinical Knowledge/Accuracy College Medicine/Accuracy Common Sense Reasoning/Accuracy Common Sense Reasoning/Accuracy Computer Security/Accuracy Econometrics/Accuracy Emotional Intelligence/Accuracy Ethics/Accuracy Fact Checking/Accuracy General Knowledge/Accuracy High School European History/Accuracy High School Geography/Accuracy High School Government and Politics/Accuracy High School Macroeconomics/Accuracy High School Microeconomics/Accuracy High School Psychology/Accuracy High School US History/Accuracy High School World History/Accuracy Human Aging/Accuracy Human Organs Senses Multiple Choice/Accuracy Human Sexuality/Accuracy Identify Odd Metapor/Accuracy Intent Recognition/Accuracy International Law/Accuracy Jurisprudence/Accuracy Logical Fallacies/Accuracy Logical Reasoning/Accuracy Logical Reasoning/Accuracy Management/Accuracy Marketing/Accuracy Mathematical Reasoning/Accuracy Mathematical Reasoning/Accuracy Medical Genetics/Accuracy Nutrition/Accuracy Odd One Out/Accuracy Philosophy/Accuracy Prehistory/Accuracy Professional Law/Accuracy Professional Medicine/Accuracy Professional Psychology/Accuracy Public Relations/Accuracy Reading Comprehension/Accuracy Reading Comprehension/Accuracy Security Studies/Accuracy Sociology/Accuracy US Foreign Policy/Accuracy Virology/Accuracy World Religions/Accuracy

Related Benchmarks

BIG-bench (Anachronisms)/Word Sense Disambiguation/Accuracy BIG-bench (Causal Judgment)/Common Sense Reasoning/Accuracy BIG-bench (Date Understanding)/Common Sense Reasoning/Accuracy BIG-bench (Disambiguation QA)/Common Sense Reasoning/Accuracy BIG-bench (Formal Fallacies Syllogisms Negation)/Logical Reasoning/Accuracy BIG-bench (Hindu Knowledge)/Memorization/Accuracy BIG-bench (Hyperbaton)/Question Answering/Accuracy BIG-bench (Known Unknowns)/Common Sense Reasoning/Accuracy BIG-bench (Logic Grid Puzzle)/Logical Reasoning/Accuracy BIG-bench (Logical Fallacy Detection)/Logical Reasoning/Accuracy BIG-bench (Logical Sequence)/Common Sense Reasoning/Accuracy BIG-bench (Movie Recommendation)/Question Answering/Accuracy BIG-bench (Navigate)/Question Answering/Accuracy BIG-bench (Novel Concepts)/Question Answering/Accuracy BIG-bench (Penguins In A Table)/Logical Reasoning/Accuracy BIG-bench (Reasoning About Colored Objects)/Logical Reasoning/Accuracy BIG-bench (Ruin Names)/Question Answering/Accuracy BIG-bench (SNARKS)/Sarcasm Detection/Accuracy BIG-bench (Sports Understanding)/Common Sense Reasoning/Accuracy BIG-bench (StrategyQA)/Logical Reasoning/Accuracy BIG-bench (Temporal Sequences)/Logical Reasoning/Accuracy BIG-bench (Winowhy)/Common Sense Reasoning/Accuracy BIG-bench-lite/Language Modelling/Accuracy Big-bench Lite/Auto Debugging/Exact string match

Statistics

Papers: 349
Benchmarks: 52

Links

Tasks

Abstract Algebra Analogical Similarity Analytic Entailment Anatomy Astronomy Auto Debugging BIG-bench Machine Learning Business Ethics Clinical Knowledge College Biology College Chemistry College Computer Science College Mathematics College Medicine College Physics Common Sense Reasoning Computer Security Conceptual Physics Crash Blossom Crass AI Dark Humor Detection Discourse Marker Prediction Econometrics Electrical Engineering Elementary Mathematics Emotional Intelligence Empirical Judgments English Proverbs Entailed Polarity Epistemic Reasoning Ethics Evaluating Information Essentiality FEVER (2-way)FEVER (3-way)Fact Checking Fantasy Reasoning Figure Of Speech Detection Formal Logic GRE Reading Comprehension General Knowledge Global Facts High School Biology High School Chemistry High School Computer Science High School European History High School Geography High School Government and Politics High School Macroeconomics High School Mathematics High School Microeconomics High School Physics High School Psychology High School Statistics High School US History High School World History Human Aging Human Organs Senses Multiple Choice Human Sexuality Identify Odd Metapor Implicatures Implicit Relations Intent Recognition International Law Irony Identification Jurisprudence LAMBADA Language Modelling Logical Args Logical Fallacies Logical Reasoning Management Marketing Mathematical Induction Mathematical Reasoning Medical Genetics Memorization Metaphor Boolean Miscellaneous Misconceptions Moral Disputes Moral Permissibility Moral Scenarios Movie Dialog Same Or Different Multi-task Language Understanding Multiple Choice Question Answering (MCQA)Natural Questions Nonsense Words Grammar Nutrition Odd One Out Philosophy Phrase Relatedness Physical Intuition Physics MC Prehistory Presuppositions As NLI Professional Accounting Professional Law Professional Medicine Professional Psychology Public Relations Question Selection RACE-h RACE-m Reading Comprehension Riddle Sense Sarcasm Detection Security Studies Sentence Ambiguity Similarities Abstraction Sociology Timedial TriviaQA US Foreign Policy Understanding Fables Virology Word Sense Disambiguation World Religions