TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation

Autor:	Yun, Taeyang, Lim, Hyunkuk, Lee, Jeonghwan, Song, Min
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language Computer Science - Machine Learning Computer Science - Sound Electrical Engineering and Systems Science - Audio and Speech Processing
Druh dokumentu:	Working Paper
Popis:	Emotion Recognition in Conversation (ERC) plays a crucial role in enabling dialogue systems to effectively respond to user requests. The emotions in a conversation can be identified by the representations from various modalities, such as audio, visual, and text. However, due to the weak contribution of non-verbal modalities to recognize emotions, multimodal ERC has always been considered a challenging task. In this paper, we propose Teacher-leading Multimodal fusion network for ERC (TelME). TelME incorporates cross-modal knowledge distillation to transfer information from a language model acting as the teacher to the non-verbal students, thereby optimizing the efficacy of the weak modalities. We then combine multimodal features using a shifting fusion approach in which student networks support the teacher. TelME achieves state-of-the-art performance in MELD, a multi-speaker conversation dataset for ERC. Finally, we demonstrate the effectiveness of our components through additional experiments. Comment: NAACL 2024 main conference
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2401.12987 Zobrazit plný text záznamu View this record from Arxiv