Normalized Contrastive Learning for Text-Video Retrieval

Autor:	Park, Yookoon, Azab, Mahmoud, Xiong, Bo, Moon, Seungwhan, Metze, Florian, Kundu, Gourab, Ahmed, Kirmani
Rok vydání:	2022
Předmět:	Computer Science - Information Retrieval Computer Science - Computer Vision and Pattern Recognition Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval. Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new state-of-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering. Comment: Published in EMNLP 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2212.11790 Zobrazit plný text záznamu View this record from Arxiv