Pythia 12B (5-shot)

Reported on 5 benchmarks across 3 tasks · 1 paper

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Natural Language Processing5 results

Question AnsweringonPIQA
Accuracy· 2023-04-03
76.7
best: 90.1 (Unicorn 11B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Common Sense ReasoningonWinoGrande
Accuracy· 2023-04-03
66.6
best: 96.1 (ST-MoE-32B 269B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Common Sense ReasoningonARC (Challenge)
Accuracy· 2023-04-03
36.8
best: 96.4 (GPT-4 (few-shot, k=25))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Common Sense ReasoningonARC (Easy)
Accuracy· 2023-04-03
71.5
best: 95.2 (ST-MoE-32B 269B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373
Coreference ResolutiononWinograd Schema Challenge
Accuracy· 2023-04-03
36.5
best: 100 (PaLM 540B (fine-tuned))
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling arXiv:2304.01373