Dual-path Convolutional Image-Text Embeddings with Instance Loss
| Autor: | Liang Zheng, Yi-Dong Shen, Zhedong Zheng, Yi Yang, Mingliang Xu, Michael Garrett |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Computer Networks and Communications Computer science Computer Vision and Pattern Recognition (cs.CV) Computer Science - Computer Vision and Pattern Recognition Initialization 02 engineering and technology Machine learning computer.software_genre Convolutional neural network Field (computer science) Ranking (information retrieval) Discriminative model 0803 Computer Software 0805 Distributed Computing 0806 Information Systems Margin (machine learning) 0202 electrical engineering electronic engineering information engineering Artificial Intelligence & Image Processing Word2vec business.industry 020207 software engineering Multimedia (cs.MM) Hardware and Architecture Feature (computer vision) 020201 artificial intelligence & image processing Artificial intelligence business computer Computer Science - Multimedia |
| Zdroj: | ACM Transactions on Multimedia Computing, Communications, and Applications. 16:1-23 |
| ISSN: | 1551-6865 1551-6857 |
| DOI: | 10.1145/3383184 |
| Popis: | Matching images and sentences demands a fine understanding of both modalities. In this paper, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image / text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss is hard for network learning, since it starts from the two heterogeneous features to build inter-modal relationship. To address this problem, we propose the instance loss which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image / text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this paper constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available. Comment: 15pages, 15 figures, 8 tables |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|