Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, Oleksandr Skurzhanskyi
In this paper, we present a simple and efficient GEC sequence tagger using a Transformer encoder. Our system is pre-trained on synthetic data and then fine-tuned in two stages: first on errorful corpora, and second on a combination of errorful and error-free parallel corpora. We design custom token-level transformations to map input tokens to target corrections. Our best single-model/ensemble GEC tagger achieves an $F_{0.5}$ of 65.3/66.5 on CoNLL-2014 (test) and $F_{0.5}$ of 72.4/73.6 on BEA-2019 (test). Its inference speed is up to 10 times as fast as a Transformer-based seq2seq GEC system. The code and trained models are publicly available.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Grammatical Error Correction | CoNLL-2014 Shared Task | F0.5 | 66.5 | Sequence tagging + token-level transformations + two-stage fine-tuning (+BERT, RoBERTa, XLNet) |
| Grammatical Error Correction | CoNLL-2014 Shared Task | Precision | 78.2 | Sequence tagging + token-level transformations + two-stage fine-tuning (+BERT, RoBERTa, XLNet) |
| Grammatical Error Correction | CoNLL-2014 Shared Task | Recall | 41.5 | Sequence tagging + token-level transformations + two-stage fine-tuning (+BERT, RoBERTa, XLNet) |
| Grammatical Error Correction | CoNLL-2014 Shared Task | F0.5 | 65.3 | Sequence tagging + token-level transformations + two-stage fine-tuning (+XLNet) |
| Grammatical Error Correction | CoNLL-2014 Shared Task | Precision | 77.5 | Sequence tagging + token-level transformations + two-stage fine-tuning (+XLNet) |
| Grammatical Error Correction | CoNLL-2014 Shared Task | Recall | 40.1 | Sequence tagging + token-level transformations + two-stage fine-tuning (+XLNet) |
| Grammatical Error Correction | BEA-2019 (test) | F0.5 | 73.7 | Sequence tagging + token-level transformations + two-stage fine-tuning (+RoBERTa, XLNet) |
| Grammatical Error Correction | BEA-2019 (test) | F0.5 | 72.4 | Sequence tagging + token-level transformations + two-stage fine-tuning (+XLNet) |