By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Autor:	Yoon, Hyungjun, Tolera, Biniyam Aschalew, Gong, Taesik, Lee, Kimin, Lee, Sung-Ju
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language Computer Science - Artificial Intelligence Computer Science - Computer Vision and Pattern Recognition Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes. Comment: Accepted to the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024) Main
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2407.10385 Zobrazit plný text záznamu View this record from Arxiv