Multimodal Sentiment Analysis based on Video and Audio Inputs

Autor:	Fernandez, Antonio, Awinat, Suzan
Rok vydání:	2024
Předmět:	Computer Science - Sound Computer Science - Artificial Intelligence Computer Science - Computer Vision and Pattern Recognition Computer Science - Multimedia Electrical Engineering and Systems Science - Audio and Speech Processing
Zdroj:	Procedia Computer Science, Volume 251, 2024, Pages 41-48, ISSN 1877-0509
Druh dokumentu:	Working Paper
DOI:	10.1016/j.procs.2024.11.082
Popis:	Despite the abundance of current researches working on the sentiment analysis from videos and audios, finding the best model that gives the highest accuracy rate is still considered a challenge for researchers in this field. The main objective of this paper is to prove the usability of emotion recognition models that take video and audio inputs. The datasets used to train the models are the CREMA-D dataset for audio and the RAVDESS dataset for video. The fine-tuned models that been used are: Facebook/wav2vec2-large for audio and the Google/vivit-b-16x2-kinetics400 for video. The avarage of the probabilities for each emotion generated by the two previous models is utilized in the decision making framework. After disparity in the results, if one of the models gets much higher accuracy, another test framework is created. The methods used are the Weighted Average method, the Confidence Level Threshold method, the Dynamic Weighting Based on Confidence method, and the Rule-Based Logic method. This limited approach gives encouraging results that make future research into these methods viable. Comment: Presented as a full paper in the 15th International Conference on Emerging Ubiquitous Systems and Pervasive Networks (EUSPN 2024) October 28-30, 2024, Leuven, Belgium
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2412.09317 Zobrazit plný text záznamu View this record from Arxiv