Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin
The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. In this paper we present a faster and simpler architecture based on a succession of convolutional layers. This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. On WMT'16 English-Romanian translation we achieve competitive accuracy to the state-of-the-art and we outperform several recently published results on the WMT'15 English-German task. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT'14 English-French translation. Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Machine Translation | IWSLT2015 German-English | BLEU score | 30.4 | Conv-LSTM (deep+pos) |
| Machine Translation | WMT2016 English-Romanian | BLEU score | 27.8 | Deep Convolutional Encoder; single-layer decoder |
| Machine Translation | WMT2016 English-Romanian | BLEU score | 27.5 | BiLSTM |
| Machine Translation | WMT2014 English-French | BLEU score | 35.7 | Deep Convolutional Encoder; single-layer decoder |