How you feelin'? Learning Emotions and Mental States in Movie Scenes

Autor:	Srivastava, Dhruv, Singh, Aditya Kumar, Tapaswi, Makarand
Rok vydání:	2023
Předmět:	Computer Science - Computer Vision and Pattern Recognition
Druh dokumentu:	Working Paper
Popis:	Movie story analysis requires understanding characters' emotions and mental states. Towards this goal, we formulate emotion understanding as predicting a diverse and multi-label set of emotions at the level of a movie scene and for each character. We propose EmoTx, a multimodal Transformer-based architecture that ingests videos, multiple characters, and dialog utterances to make joint predictions. By leveraging annotations from the MovieGraphs dataset, we aim to predict classic emotions (e.g. happy, angry) and other mental states (e.g. honest, helpful). We conduct experiments on the most frequently occurring 10 and 25 labels, and a mapping that clusters 181 labels to 26. Ablation studies and comparison against adapted state-of-the-art emotion recognition approaches shows the effectiveness of EmoTx. Analyzing EmoTx's self-attention scores reveals that expressive emotions often look at character tokens while other mental states rely on video and dialog cues. Comment: CVPR 2023. Project Page: https://katha-ai.github.io/projects/emotx/
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2304.05634 Zobrazit plný text záznamu View this record from Arxiv