Bridging auditory perception and natural language processing with semantically informed deep neural networks.

Autor: Esposito M; Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands. m.esposito@maastrichtuniversity.nl., Valente G; Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands., Plasencia-Calaña Y; BISS Institute, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands., Dumontier M; Institute of Data Science, Maastricht University, Maastricht, The Netherlands., Giordano BL; Institut des Neurosciences de La Timone, CNRS UMR 7289-Université Aix-Marseille, Marseille, France., Formisano E; Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands. e.formisano@maastrichtuniversity.nl.; BISS Institute, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands. e.formisano@maastrichtuniversity.nl.
Jazyk: angličtina
Zdroj: Scientific reports [Sci Rep] 2024 Sep 09; Vol. 14 (1), pp. 20994. Date of Electronic Publication: 2024 Sep 09.
DOI: 10.1038/s41598-024-71693-9
Abstrakt: Sound recognition is effortless for humans but poses a significant challenge for artificial hearing systems. Deep neural networks (DNNs), especially convolutional neural networks (CNNs), have recently surpassed traditional machine learning in sound classification. However, current DNNs map sounds to labels using binary categorical variables, neglecting the semantic relations between labels. Cognitive neuroscience research suggests that human listeners exploit such semantic information besides acoustic cues. Hence, our hypothesis is that incorporating semantic information improves DNN's sound recognition performance, emulating human behaviour. In our approach, sound recognition is framed as a regression problem, with CNNs trained to map spectrograms to continuous semantic representations from NLP models (Word2Vec, BERT, and CLAP text encoder). Two DNN types were trained: semDNN with continuous embeddings and catDNN with categorical labels, both with a dataset extracted from a collection of 388,211 sounds enriched with semantic descriptions. Evaluations across four external datasets, confirmed the superiority of semantic labeling from semDNN compared to catDNN, preserving higher-level relations. Importantly, an analysis of human similarity ratings for natural sounds, showed that semDNN approximated human listener behaviour better than catDNN, other DNNs, and NLP models. Our work contributes to understanding the role of semantics in sound recognition, bridging the gap between artificial systems and human auditory perception.
(© 2024. The Author(s).)
Databáze: MEDLINE
Externí odkaz:
Nepřihlášeným uživatelům se plný text nezobrazuje