Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning

Autor:	Liao, Feng-Ting, Chan, Yung-Chieh, Chen, Yi-Chang, Hsu, Chan-Jan, Shiu, Da-shan
Rok vydání:	2023
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	In this work, we propose a method to create domain-sensitive speech recognition models that utilize textual domain information by conditioning its generation on a given text prompt. This is accomplished by fine-tuning a pre-trained, end-to-end model (Whisper) to learn from demonstrations with prompt examples. We show that this ability can be generalized to different domains and even various prompt contexts, with our model gaining a Word Error Rate (WER) reduction of up to 33% on unseen datasets from various domains, such as medical conversation, air traffic control communication, and financial meetings. Considering the limited availability of audio-transcript pair data, we further extend our method to text-only fine-tuning to achieve domain sensitivity as well as domain adaptation. We demonstrate that our text-only fine-tuned model can also attend to various prompt contexts, with the model reaching the most WER reduction of 29% on the medical conversation dataset. Comment: F-T Liao and Y-C Chan contributed equally; paper accepted to ASRU2023; code and model weights available in https://github.com/mtkresearch/clairaudience
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2307.10274 Zobrazit plný text záznamu View this record from Arxiv