SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency

Autor:	Dharur, Sameer, Tendulkar, Purva, Batra, Dhruv, Parikh, Devi, Selvaraju, Ramprasaath R.
Rok vydání:	2020
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world -- they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the higher level question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub-questions higher than irrelevant questions for an pair. We show that SOrT improves model consistency by upto 6.5% points over existing baselines, while also improving visual grounding. Comment: Accepted to the NeurIPS 2020 workshop on Interpretable Inductive Biases and Physically Structured Learning
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2010.10038 Zobrazit plný text záznamu View this record from Arxiv