Enhancing Vision Models for Text-Heavy Content Understanding and Interaction

Autor:	TG, Adithya, SK, Adithya, Bharadwaj, Abhinav R, HA, Abhiram, Narayan, Surabhi
Rok vydání:	2024
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	Interacting and understanding with text heavy visual content with multiple images is a major challenge for traditional vision models. This paper is on enhancing vision models' capability to comprehend or understand and learn from images containing a huge amount of textual information from the likes of textbooks and research papers which contain multiple images like graphs, etc and tables in them with different types of axes and scales. The approach involves dataset preprocessing, fine tuning which is by using instructional oriented data and evaluation. We also built a visual chat application integrating CLIP for image encoding and a model from the Massive Text Embedding Benchmark which is developed to consider both textual and visual inputs. An accuracy of 96.71% was obtained. The aim of the project is to increase and also enhance the advance vision models' capabilities in understanding complex visual textual data interconnected data, contributing to multimodal AI. Comment: 5 pages, 4 figures (including 1 graph)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2405.20906 Zobrazit plný text záznamu View this record from Arxiv