Learning Sound Localization Better From Semantically Similar Samples

Autor:	Senocak, Arda, Ryu, Hyeonggon, Kim, Junsik, Kweon, In So
Rok vydání:	2022
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Sound Electrical Engineering and Systems Science - Audio and Speech Processing Electrical Engineering and Systems Science - Image and Video Processing
Druh dokumentu:	Working Paper
Popis:	The objective of this work is to localize the sound sources in visual scenes. Existing audio-visual works employ contrastive learning by assigning corresponding audio-visual pairs from the same source as positives while randomly mismatched pairs as negatives. However, these negative pairs may contain semantically matched audio-visual information. Thus, these semantically correlated pairs, "hard positives", are mistakenly grouped as negatives. Our key contribution is showing that hard positives can give similar response maps to the corresponding pairs. Our approach incorporates these hard positives by adding their response maps into a contrastive learning objective directly. We demonstrate the effectiveness of our approach on VGG-SS and SoundNet-Flickr test sets, showing favorable performance to the state-of-the-art methods. Comment: Accepted to ICASSP 2022. SOTA performance in Audio-Visual Sound Localization. 5 Pages
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2202.03007 Zobrazit plný text záznamu View this record from Arxiv