Challenge Training to Simulate Inference in Machine Translation
| Autor: | Jie Zhou, Wenjie Lu, Leiying Zhou, Gongshen Liu, Quanhai Zhang |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Machine translation
Computer science business.industry Maximum likelihood 05 social sciences Inference 010501 environmental sciences computer.software_genre Machine learning 01 natural sciences 0502 economics and business Leverage (statistics) Artificial intelligence 050207 economics business computer Decoding methods 0105 earth and related environmental sciences |
| Zdroj: | IJCNN |
| DOI: | 10.1109/ijcnn48605.2020.9206915 |
| Popis: | Despite much success has been achieved, neural machine translation (NMT) suffers from exposure bias and evaluation discrepancy. To be specific, the generation inconsistency between the training and inference process further causes error accumulation and distribution disparity. Furthermore, NMT models are generally optimized on word-level cross-entropy loss function but evaluated by sentence-level metrics. This evaluation-level mismatch may mislead the promotion of translation performance. To address these two drawbacks, we propose to challenge training to gradually simulate inference. Namely, the decoder is fed with inferred words rather than ground truth words during training with a dynamic probability. To ensure accuracy and integrity, we adopt alignment and tailoring on the inferred words. Therefore, these words can leverage inferred information to help improve the training process. As for the dynamic simulation, we define a novel loss-sensitive probability that can sense the converge of training and finetune itself in turn. Experimental results on IWSLT 2016 German-English and WMT 2019 English-Chinese datasets demonstrate that our methodology can significantly improve translation quality. The approach of alignment and tailoring outperforms previous works. Meanwhile, the proposed loss-sensitive sampling is also useful for other state-of-the-art scheduled sampling methods to achieve further promotion. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|