StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis

Autor:	Chen, Zhiyong, Li, Xinnuo, Ai, Zhiqi, Xu, Shugong
Rok vydání:	2024
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing Computer Science - Sound
Zdroj:	The 7th Chinese Conference on Pattern Recognition and Computer Vision PRCV 2024
Druh dokumentu:	Working Paper
Popis:	We introduce StyleFusion-TTS, a prompt and/or audio referenced, style and speaker-controllable, zero-shot text-to-speech (TTS) synthesis system designed to enhance the editability and naturalness of current research literature. We propose a general front-end encoder as a compact and effective module to utilize multimodal inputs including text prompts, audio references, and speaker timbre references in a fully zero-shot manner and produce disentangled style and speaker control embeddings. Our novel approach also leverages a hierarchical conformer structure for the fusion of style and speaker control embeddings, aiming to achieve optimal feature fusion within the current advanced TTS architecture. StyleFusion-TTS is evaluated through multiple metrics, both subjectively and objectively. The system shows promising performance across our evaluations, suggesting its potential to contribute to the advancement of the field of zero-shot text-to-speech synthesis. Comment: The 7th Chinese Conference on Pattern Recognition and Computer Vision PRCV 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2409.15741 Zobrazit plný text záznamu View this record from Arxiv