Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Autor:	Li, Qifei, Gao, Yingming, Wen, Yuhua, Wang, Cong, Li, Ya
Rok vydání:	2024
Předmět:	Computer Science - Multimedia Computer Science - Artificial Intelligence Computer Science - Computer Vision and Pattern Recognition Computer Science - Sound
Druh dokumentu:	Working Paper
Popis:	To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after alignment, called Foal-Net. The framework is designed to enhance the effectiveness of modality fusion and includes two auxiliary tasks: audio-video emotion alignment (AVEL) and cross-modal emotion label matching (MEM). First, AVEL achieves alignment of emotional information in audio-video representations through contrastive learning. Then, a modal fusion network integrates the aligned features. Meanwhile, MEM assesses whether the emotions of the current sample pair are the same, providing assistance for modal information fusion and guiding the model to focus more on emotional information. The experimental results conducted on IEMOCAP corpus show that Foal-Net outperforms the state-of-the-art methods and emotion alignment is necessary before modal fusion. Comment: The paper has been accepted by INTERSPEECH 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2408.09438 Zobrazit plný text záznamu View this record from Arxiv