AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description

Autor:	Xie, Junyu, Han, Tengda, Bain, Max, Nagrani, Arsha, Varol, Gül, Xie, Weidi, Zisserman, Andrew
Rok vydání:	2024
Předmět:	Computer Science - Computer Vision and Pattern Recognition
Druh dokumentu:	Working Paper
Popis:	Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs), and develop visual and text prompting strategies for this task. Our contributions are three-fold: (i) We demonstrate that a VLM can successfully name and refer to characters if directly prompted with character information through visual indications without requiring any fine-tuning; (ii) A two-stage process is developed to generate ADs, with the first stage asking the VLM to comprehensively describe the video, followed by a second stage utilising a LLM to summarise dense textual information into one succinct AD sentence; (iii) A new dataset for TV audio description is formulated. Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores. Comment: Project Page: https://www.robots.ox.ac.uk/~vgg/research/autoad-zero/
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2407.15850 Zobrazit plný text záznamu View this record from Arxiv