Branch-Train-MiX 4x7B (sampling top-1 experts)

Reported on 2 benchmarks across 2 tasks · 1 paper

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Methodology2 results

Transfer LearningonMML
Average (%)· 2024-03-12
53.2
best: 87 (GPT-4 o1(300b))
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM arXiv:2403.07816
Multi-Task LearningonMML
Average (%)· 2024-03-12
53.2
best: 87 (GPT-4 o1(300b))
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM arXiv:2403.07816