GEM: A General Evaluation Benchmark for Multimodal Tasks
| Autor: | Nan Duan, Lei Ji, Edward Cui, Ming Zhong, Huaishao Luo, Taroon Bharti, Lin Su, Chenfei Wu, Arun Sacheti, Yongfei Liu |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Computation and Language business.industry Computer science Computer Vision and Pattern Recognition (cs.CV) Computer Science - Computer Vision and Pattern Recognition Machine learning computer.software_genre Multimedia (cs.MM) Benchmark (computing) Code (cryptography) Artificial intelligence business Baseline (configuration management) computer Computation and Language (cs.CL) Computer Science - Multimedia Natural language |
| Zdroj: | ACL/IJCNLP (Findings) |
| DOI: | 10.48550/arxiv.2106.09889 |
| Popis: | In this paper, we present GEM as a General Evaluation benchmark for Multimodal tasks. Different from existing datasets such as GLUE, SuperGLUE, XGLUE and XTREME that mainly focus on natural language tasks, GEM is a large-scale vision-language benchmark, which consists of GEM-I for image-language tasks and GEM-V for video-language tasks. Comparing with existing multimodal datasets such as MSCOCO and Flicker30K for image-language tasks, YouCook2 and MSR-VTT for video-language tasks, GEM is not only the largest vision-language dataset covering image-language tasks and video-language tasks at the same time, but also labeled in multiple languages. We also provide two baseline models for this benchmark. We will release the dataset, code and baseline models, aiming to advance the development of multilingual multimodal research. Comment: Accepted by Findings of ACL 2021 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|