Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
| Autor: | Furu Wei, Chunyuan Li, Pengchuan Zhang, Yejin Choi, Lijuan Wang, Jianfeng Gao, Li Dong, Houdong Hu, Lei Zhang, Xi Yin, Xiaowei Hu, Xiujun Li |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Downstream (software development)
Computer science business.industry 02 engineering and technology 010501 environmental sciences Object (computer science) computer.software_genre Semantics 01 natural sciences Image (mathematics) 0202 electrical engineering electronic engineering information engineering Code (cryptography) 020201 artificial intelligence & image processing Artificial intelligence business computer Natural language processing 0105 earth and related environmental sciences |
| Zdroj: | Computer Vision – ECCV 2020 ISBN: 9783030585761 ECCV (30) |
| Popis: | Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks (The code and pre-trained models are released: https://github.com/microsoft/Oscar). |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|