Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval
| Autor: | Weifeng Zhang, Zengchang Qin, Jianlong Tan, Yue Hu, Jing Yu, Qi Wu, Yuhang Lu |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Relation (database)
business.industry Computer science 02 engineering and technology Visual reasoning Semantics computer.software_genre Computer Science Applications Visualization Knowledge extraction Signal Processing 0202 electrical engineering electronic engineering information engineering Media Technology Task analysis Question answering 020201 artificial intelligence & image processing Artificial intelligence Electrical and Electronic Engineering business Representation (mathematics) computer Natural language processing |
| Zdroj: | IEEE Transactions on Multimedia. 22:3196-3209 |
| ISSN: | 1941-0077 1520-9210 |
| DOI: | 10.1109/tmm.2020.2972830 |
| Popis: | Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|