AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model

Autor:	Yeo, Jeong Hun, Kim, Minsu, Choi, Jeongsoo, Kim, Dae Hoe, Ro, Yong Man
Rok vydání:	2023
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Multimedia Electrical Engineering and Systems Science - Audio and Speech Processing Electrical Engineering and Systems Science - Image and Video Processing
Druh dokumentu:	Working Paper
Popis:	Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements. VSR is regarded as a challenging task because of the insufficient information on lip movements. In this paper, we propose an Audio Knowledge empowered Visual Speech Recognition framework (AKVSR) to complement the insufficient speech information of visual modality by using audio modality. Different from the previous methods, the proposed AKVSR 1) utilizes rich audio knowledge encoded by a large-scale pretrained audio model, 2) saves the linguistic information of audio knowledge in compact audio memory by discarding the non-linguistic information from the audio through quantization, and 3) includes Audio Bridging Module which can find the best-matched audio features from the compact audio memory, which makes our training possible without audio inputs, once after the compact audio memory is composed. We validate the effectiveness of the proposed method through extensive experiments, and achieve new state-of-the-art performances on the widely-used LRS3 dataset. Comment: Accepted by IEEE Transactions on Multimedia
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2308.07593 Zobrazit plný text záznamu View this record from Arxiv