Peter Izsak, Moshe Berchansky, Omer Levy
While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Question Answering | Quora Question Pairs | Accuracy | 70.7 | 24hBERT |
| Natural Language Inference | QNLI | Accuracy | 90.6 | 24hBERT |
| Natural Language Inference | MultiNLI | Matched | 84.4 | 24hBERT |
| Natural Language Inference | MultiNLI | Mismatched | 83.8 | 24hBERT |
| Semantic Textual Similarity | STS Benchmark | Pearson Correlation | 0.82 | 24hBERT |
| Sentiment Analysis | SST-2 Binary classification | Accuracy | 93 | 24hBERT |
| Linguistic Acceptability | CoLA | Accuracy | 57.1 | 24hBERT |