Pythia 12B (0-shot)

Reported on 6 benchmarks across 4 tasks · 1 paper

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Natural Language Processing5 results

Question AnsweringonPIQA
Accuracy· 2023-04-03
76
best: 90.1 (Unicorn 11B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Common Sense ReasoningonWinoGrande
Accuracy· 2023-04-03
63.9
best: 96.1 (ST-MoE-32B 269B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Common Sense ReasoningonARC (Challenge)
Accuracy· 2023-04-03
31.8
best: 96.4 (GPT-4 (few-shot, k=25))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Common Sense ReasoningonARC (Easy)
Accuracy· 2023-04-03
70.2
best: 95.2 (ST-MoE-32B 269B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Coreference ResolutiononWinograd Schema Challenge
Accuracy· 2023-04-03
54.8
best: 100 (PaLM 540B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373

Medical1 result

Language ModellingonLAMBADA
Accuracy· 2023-04-03
70.46
best: 89.7 (PaLM-540B (Few-Shot))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373