Cross-lingual Prosody Transfer for Expressive Machine Dubbing

Autor:	Swiatkowski, Jakub, Wang, Duo, Babianski, Mikolaj, Tobing, Patrick Lumban, Vipperla, Ravichander, Pollet, Vincent
Rok vydání:	2023
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing
Druh dokumentu:	Working Paper
Popis:	Prosody transfer is well-studied in the context of expressive speech synthesis. Cross-lingual prosody transfer, however, is challenging and has been under-explored to date. In this paper, we present a novel solution to learn prosody representations that are transferable across languages and speakers for machine dubbing of expressive multimedia contents. Multimedia contents often contain field recordings. To enable prosody transfer from noisy audios, we introduce a novel noise modelling module that disentangles noise conditioning from prosody conditioning, and thereby gains independent control of noise levels in the synthesised speech. We augment noisy training data with clean data to improve the ability of the model to map the denoised reference audio to clean speech. Our proposed system can generate speech with context-matching prosody and closes the gap between a strong baseline and human expressive dialogs by 11.2%. Comment: Submitted to INTERSPEECH
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2306.11658 Zobrazit plný text záznamu View this record from Arxiv