Testing Untestable Neural Machine Translation: An Industrial Case
| Autor: | Pinjia He, Wenyu Wang, Qinsong Zeng, Tao Xie, Wei Yang, Dian Liu, Yuetang Deng, Wujie Zheng, Changrong Zhang |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
FOS: Computer and information sciences
Information privacy Computer Science - Computation and Language Machine translation Computer Science - Artificial Intelligence business.industry Computer science Mobile computing System testing 020207 software engineering 02 engineering and technology computer.software_genre Software Engineering (cs.SE) Computer Science - Software Engineering Artificial Intelligence (cs.AI) 020204 information systems 0202 electrical engineering electronic engineering information engineering Task analysis Language translation Software engineering business computer Computation and Language (cs.CL) |
| Zdroj: | ICSE (Companion Volume) |
| DOI: | 10.48550/arxiv.1807.02340 |
| Popis: | Neural Machine Translation (NMT) has been widely adopted recently due to its advantages compared with the traditional Statistical Machine Translation (SMT). However, an NMT system still often produces translation failures due to the complexity of natural language and sophistication in designing neural networks. While in-house black-box system testing based on reference translations (i.e., examples of valid translations) has been a common practice for NMT quality assurance, an increasingly critical industrial practice, named in-vivo testing, exposes unseen types or instances of translation failures when real users are using a deployed industrial NMT system. To fill the gap of lacking test oracle for in-vivo testing of an NMT system, in this paper, we propose a new approach for automatically identifying translation failures, without requiring reference translations for a translation task; our approach can directly serve as a test oracle for in-vivo testing. Our approach focuses on properties of natural language translation that can be checked systematically and uses information from both the test inputs (i.e., the texts to be translated) and the test outputs (i.e., the translations under inspection) of the NMT system. Our evaluation conducted on real-world datasets shows that our approach can effectively detect targeted property violations as translation failures. Our experiences on deploying our approach in both production and development environments of WeChat (a messenger app with over one billion monthly active users) demonstrate high effectiveness of our approach along with high industry impact. Comment: 10 pages |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|