Exploring Unsupervised Pretraining Objectives for Machine Translation
| Autor: | Ivan Titov, Alexandra Birch, Barry Haddow, Christos Baziotis |
|---|---|
| Přispěvatelé: | Zong, Chengqing, Xia, Fei, Li, Wenjie, Navigli, Roberto |
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Předmět: |
Masking (art)
FOS: Computer and information sciences Computer Science - Computation and Language Machine translation business.industry Computer science Contrast (statistics) Context (language use) computer.software_genre Machine learning ENCODE Artificial intelligence business computer Computation and Language (cs.CL) |
| Zdroj: | ACL/IJCNLP (Findings) Baziotis, C, Titov, I, Birch, A & Haddow, B 2021, Exploring Unsupervised Pretraining Objectives for Machine Translation . in C Zong, F Xia, W Li & R Navigli (eds), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . pp. 2956-2971, The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Bangkok, Thailand, 1/08/21 . https://doi.org/10.18653/v1/2021.findings-acl.261 |
| Popis: | Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. In this work, we systematically compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English$\leftrightarrow$German, English$\leftrightarrow$Nepali and English$\leftrightarrow$Sinhala monolingual data, and evaluate them on NMT. In (semi-) supervised NMT, varying the pretraining objective leads to surprisingly small differences in the finetuned performance, whereas unsupervised NMT is much more sensitive to it. To understand these results, we thoroughly study the pretrained models using a series of probes and verify that they encode and use information in different ways. We conclude that finetuning on parallel data is mostly sensitive to few properties that are shared by most models, such as a strong decoder, in contrast to unsupervised NMT that also requires models with strong cross-lingual abilities. Findings of ACL 2021 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|