Speech foundation models in healthcare: Effect of layer selection on pathological speech feature prediction

Autor:	Wiepert, Daniela A., Utianski, Rene L., Duffy, Joseph R., Stricker, John L., Barnard, Leland R., Jones, David T., Botha, Hugo
Rok vydání:	2024
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Accurately extracting clinical information from speech is critical to the diagnosis and treatment of many neurological conditions. As such, there is interest in leveraging AI for automatic, objective assessments of clinical speech to facilitate diagnosis and treatment of speech disorders. We explore transfer learning using foundation models, focusing on the impact of layer selection for the downstream task of predicting pathological speech features. We find that selecting an optimal layer can greatly improve performance (~15.8% increase in balanced accuracy per feature as compared to worst layer, ~13.6% increase as compared to final layer), though the best layer varies by predicted feature and does not always generalize well to unseen data. A learned weighted sum offers comparable performance to the average best layer in-distribution (only ~1.2% lower) and had strong generalization for out-of-distribution data (only 1.5% lower than the average best layer). Comment: Accepted to INTERSPEECH 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2402.01796 Zobrazit plný text záznamu View this record from Arxiv