Prefix tuning for automated audio captioning
| Autor: | Minkyu Kim, Kim Sung-Bin, Tae-Hyun Oh |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2023 |
| Předmět: |
FOS: Computer and information sciences
Sound (cs.SD) Audio and Speech Processing (eess.AS) FOS: Electrical engineering electronic engineering information engineering Computer Science - Multimedia Computer Science - Sound Multimedia (cs.MM) Electrical Engineering and Systems Science - Audio and Speech Processing |
| Popis: | Audio captioning aims to generate text descriptions from environmental sounds. One challenge of audio captioning is the difficulty of the generalization due to the lack of audio-text paired training data. In this work, we propose a simple yet effective method of dealing with small-scaled datasets by leveraging a pre-trained language model. We keep the language model frozen to maintain the expressivity for text generation, and we only learn to extract global and temporal features from the input audio. To bridge a modality gap between the audio features and the language model, we employ mapping networks that translate audio features to the continuous vectors the language model can understand, called prefixes. We evaluate our proposed method on the Clotho and AudioCaps dataset and show our method outperforms prior arts in diverse experimental settings. ICASSP 2023 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|