Multi-View Spectrogram Transformer for Respiratory Sound Classification

Autor:	He, Wentao, Yan, Yuchen, Ren, Jianfeng, Bai, Ruibin, Jiang, Xudong
Rok vydání:	2023
Předmět:	Computer Science - Sound Computer Science - Computer Vision and Pattern Recognition Electrical Engineering and Systems Science - Audio and Speech Processing
Druh dokumentu:	Working Paper
Popis:	Deep neural networks have been applied to audio spectrograms for respiratory sound classification. Existing models often treat the spectrogram as a synthetic image while overlooking its physical characteristics. In this paper, a Multi-View Spectrogram Transformer (MVST) is proposed to embed different views of time-frequency characteristics into the vision transformer. Specifically, the proposed MVST splits the mel-spectrogram into different sized patches, representing the multi-view acoustic elements of a respiratory sound. These patches and positional embeddings are then fed into transformer encoders to extract the attentional information among patches through a self-attention mechanism. Finally, a gated fusion scheme is designed to automatically weigh the multi-view features to highlight the best one in a specific scenario. Experimental results on the ICBHI dataset demonstrate that the proposed MVST significantly outperforms state-of-the-art methods for classifying respiratory sounds. Comment: The paper was published at ICASSP 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2311.09655 Zobrazit plný text záznamu View this record from Arxiv