Detecting Failures of Neural Machine Translation in the Absence of Reference Translations
| Autor: | Tao Xie, Wei Yang, Wenyu Wang, Wujie Zheng, Pinjia He, Qinsong Zeng, Changrong Zhang, Dian Liu, Yuetang Deng |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Machine translation
Property (programming) business.industry Computer science media_common.quotation_subject System testing 020207 software engineering 02 engineering and technology computer.software_genre Machine learning Oracle Task (project management) 020204 information systems Scalability 0202 electrical engineering electronic engineering information engineering Quality (business) Artificial intelligence business computer Natural language media_common |
| Zdroj: | DSN (Industry Track) |
| DOI: | 10.1109/dsn-industry.2019.00007 |
| Popis: | Despite getting widely adopted recently, a Neural Machine Translation (NMT) system is often found to produce translation failures in the outputs. Developers have been relying on in-house system testing for quality assurance of NMT. This testing methodology requires human-constructed reference translations as the ground truth (test oracle) for example natural language inputs. The testing methodology has shown benefits of quickly enhancing an NMT system in early development stages. However, in industrial settings, it is desirable to detect translation failures without reliance on reference translations for enabling further improvements on translation quality in both industrial development and production environments. Aiming for a practical and scalable solution to such demand in the industrial settings, in this paper, we propose a new approach for automatically identifying translation failures without requiring reference translations for a translation task. Our approach focuses on a property of natural language translation that can be checked systematically by using information from both the test inputs (i.e., the texts to be translated) and the test outputs (i.e., the translations under inspection) of the NMT system. Our evaluation conducted on real-world datasets shows that our approach can effectively detect property violations as translation failures. By deploying our approach in the translation service of WeChat (a messenger app with more than one billion monthly active users), we show that our approach is both practical and scalable in the industrial settings. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|