Temporal Knowledge Distillation for On-device Audio Classification

Autor:	Choi, Kwanghee, Kersner, Martin, Morton, Jacob, Chang, Buru
Rok vydání:	2021
Předmět:	Computer Science - Sound Computer Science - Machine Learning Electrical Engineering and Systems Science - Audio and Speech Processing
Druh dokumentu:	Working Paper
Popis:	Improving the performance of on-device audio classification models remains a challenge given the computational limits of the mobile environment. Many studies leverage knowledge distillation to boost predictive performance by transferring the knowledge from large models to on-device models. However, most lack a mechanism to distill the essence of the temporal information, which is crucial to audio classification tasks, or similar architecture is often required. In this paper, we propose a new knowledge distillation method designed to incorporate the temporal knowledge embedded in attention weights of large transformer-based models into on-device models. Our distillation method is applicable to various types of architectures, including the non-attention-based architectures such as CNNs or RNNs, while retaining the original network architecture during inference. Through extensive experiments on both an audio event detection dataset and a noisy keyword spotting dataset, we show that our proposed method improves the predictive performance across diverse on-device architectures. Comment: ICASSP 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2110.14131 Zobrazit plný text záznamu View this record from Arxiv