What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models

Autor:	Çoban, Enis Berk, Mandel, Michael I., Devaney, Johanna
Rok vydání:	2024
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing Computer Science - Computation and Language Computer Science - Sound
Druh dokumentu:	Working Paper
Popis:	Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities, notably in connecting ideas and adhering to logical rules to solve problems. These models have evolved to accommodate various data modalities, including sound and images, known as multimodal LLMs (MLLMs), which are capable of describing images or sound recordings. Previous work has demonstrated that when the LLM component in MLLMs is frozen, the audio or visual encoder serves to caption the sound or image input facilitating text-based reasoning with the LLM component. We are interested in using the LLM's reasoning capabilities in order to facilitate classification. In this paper, we demonstrate through a captioning/classification experiment that an audio MLLM cannot fully leverage its LLM's text-based reasoning when generating audio captions. We also consider how this may be due to MLLMs separately representing auditory and textual information such that it severs the reasoning pathway from the LLM to the audio encoder. Comment: 9 pages
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2406.04615 Zobrazit plný text záznamu View this record from Arxiv