MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Autor:	Shaikh, Muhammad Bilal, Chai, Douglas, Islam, Syed Mohammed Shamsul, Akhtar, Naveed
Rok vydání:	2022
Předmět:	Computer Science - Computer Vision and Pattern Recognition I.2.10 I.5.4 I.5.2
Zdroj:	2022 IEEE International Conference on Visual Communications and Image Processing (VCIP)
Druh dokumentu:	Working Paper
DOI:	10.1109/VCIP56404.2022.10008833
Popis:	Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNNs can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset. Comment: Peer reviewed & accepted at IEEE VCIP 2022 (http://www.vcip2022.org/)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2209.04780 Zobrazit plný text záznamu View this record from Arxiv