Audio-Visual Speech Enhancement with Score-Based Generative Models

Autor:	Richter, Julius, Frintrop, Simone, Gerkmann, Timo
Rok vydání:	2023
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	This paper introduces an audio-visual speech enhancement system that leverages score-based generative models, also known as diffusion models, conditioned on visual information. In particular, we exploit audio-visual embeddings obtained from a self-super\-vised learning model that has been fine-tuned on lipreading. The layer-wise features of its transformer-based encoder are aggregated, time-aligned, and incorporated into the noise conditional score network. Experimental evaluations show that the proposed audio-visual speech enhancement system yields improved speech quality and reduces generative artifacts such as phonetic confusions with respect to the audio-only equivalent. The latter is supported by the word error rate of a downstream automatic speech recognition model, which decreases noticeably, especially at low input signal-to-noise ratios. Comment: Submitted to ITG Conference on Speech Communication
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2306.01432 Zobrazit plný text záznamu View this record from Arxiv