On the use of BERT for Neural Machine Translation

Autor:	Clinchant, Stéphane, Jung, Kweon Woo, Nikoulina, Vassilina
Rok vydání:	2019
Předmět:	Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness. Comment: 10 pages
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1909.12744 Zobrazit plný text záznamu View this record from Arxiv