Latent Alignment of Procedural Concepts in Multimodal Recipes
| Autor: | Hossein Rajaby Faghihi, Roshanak Mirzaee, Parisa Kordjamshidi, Sudarshan Paliwal |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Computation and Language Computer Science - Artificial Intelligence business.industry Computer science I.2.7 Matrix (music) Disjoint sets Space (commercial competition) computer.software_genre Task (project management) Artificial Intelligence (cs.AI) Reading comprehension Artificial intelligence business computer Computation and Language (cs.CL) Natural language processing |
| DOI: | 10.48550/arxiv.2101.04727 |
| Popis: | We propose a novel alignment mechanism to deal with procedural reasoning on a newly released multimodal QA dataset, named RecipeQA. Our model is solving the textual cloze task which is a reading comprehension on a recipe containing images and instructions. We exploit the power of attention networks, cross-modal representations, and a latent alignment space between instructions and candidate answers to solve the problem. We introduce constrained max-pooling which refines the max-pooling operation on the alignment matrix to impose disjoint constraints among the outputs of the model. Our evaluation result indicates a 19\% improvement over the baselines. Comment: Published in ALVR 2020, a workshop in ACL 2020 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|