Tasks SotA Datasets Papers Methods Submit About

Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable Benchmarks All SotA Datasets Papers Methods

Community

Submit Results About

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

MML

Massive Multitask Language Understanding

TextsCustomIntroduced 2020-09-07

MMLU (Massive Multitask Language Understanding) is a new benchmark designed to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more specialized areas like law and ethics. The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots.

Image source: https://arxiv.org/pdf/2009.03300v3.pdf

Benchmarks

Multi-Task Learning/Average (%)Question Answering/Accuracy Transfer Learning/Average (%)

Related Benchmarks

MMLU (5-Shot)/Multi-Task Learning/MMLU (5-shot)MMLU (5-Shot)/Transfer Learning/MMLU (5-shot)MMLU (Abstract Algebra)/Question Answering/Accuracy MMLU (Anatomy)/Question Answering/Accuracy MMLU (Astronomy)/Question Answering/Accuracy MMLU (Clinical Knowledge)/Question Answering/Accuracy MMLU (College Biology)/Question Answering/Accuracy MMLU (College Chemistry)/Question Answering/Accuracy MMLU (College Computer Science)/Question Answering/Accuracy MMLU (College Mathematics)/Question Answering/Accuracy MMLU (College Medicine)/Question Answering/Accuracy MMLU (College Physics)/Question Answering/Accuracy MMLU (Econometrics)/Question Answering/Accuracy MMLU (Electrical Engineer)/Question Answering/Accuracy MMLU (Elementary Mathematics)/Question Answering/Accuracy MMLU (Formal Logic)/Question Answering/Accuracy MMLU (High School Biology)/Question Answering/Accuracy MMLU (High School Chemistry)/Question Answering/Accuracy MMLU (High School Computer Science)/Question Answering/Accuracy MMLU (High School Mathematics)/Question Answering/Accuracy MMLU (High School Physics)/Question Answering/Accuracy MMLU (High School Statistics)/Question Answering/Accuracy MMLU (Machine Learning)/Question Answering/Accuracy MMLU (Medical Genetics)/Question Answering/Accuracy MMLU (Professional medicine)/Question Answering/Accuracy MMLU-Pro/MMLU/0-shot MRR

Statistics

Papers: 1,922
Benchmarks: 3

Links

Tasks

Multi-Task Learning Multi-task Language Understanding Multiple Choice Question Answering (MCQA)Natural Language Understanding Question Answering Single Choice Question Text Generation Transfer Learning