Data-Efficient Sensor Upgrade Path Using Knowledge Distillation
| Autor: | Cedric De Boom, Tim Verbelen, Bert Vankeirsbilck, Bart Diricx, Pieter Simoens, Pieter Van Molle, Jonas De Vylder, Bart Dhoedt |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Předmět: |
Technology and Engineering
Computer science MELANOMA TP1-1185 DIAGNOSIS Machine learning computer.software_genre Skin Diseases Biochemistry Article Analytical Chemistry sensor multispectral imaging Humans Leverage (statistics) sensor upgrade Electrical and Electronic Engineering Instrumentation Hyperparameter Modality (human–computer interaction) Contextual image classification Artificial neural network business.industry Chemical technology Deep learning deep learning Atomic and Molecular Physics and Optics knowledge distillation cross-modal distillation Test set skin lesion classification upgrade Neural Networks Computer Artificial intelligence business computer MNIST database |
| Zdroj: | Sensors Volume 21 Issue 19 Sensors, Vol 21, Iss 6523, p 6523 (2021) Sensors (Basel, Switzerland) SENSORS |
| ISSN: | 1424-8220 |
| DOI: | 10.3390/s21196523 |
| Popis: | Deep neural networks have achieved state-of-the-art performance in image classification. Due to this success, deep learning is now also being applied to other data modalities such as multispectral images, lidar and radar data. However, successfully training a deep neural network requires a large reddataset. Therefore, transitioning to a new sensor modality (e.g., from regular camera images to multispectral camera images) might result in a drop in performance, due to the limited availability of data in the new modality. This might hinder the adoption rate and time to market for new sensor technologies. In this paper, we present an approach to leverage the knowledge of a teacher network, that was trained using the original data modality, to improve the performance of a student network on a new data modality: a technique known in literature as knowledge distillation. By applying knowledge distillation to the problem of sensor transition, we can greatly speed up this process. We validate this approach using a multimodal version of the MNIST dataset. Especially when little data is available in the new modality (i.e., 10 images), training with additional teacher supervision results in increased performance, with the student network scoring a test set accuracy of 0.77, compared to an accuracy of 0.37 for the baseline. We also explore two extensions to the default method of knowledge distillation, which we evaluate on a multimodal version of the CIFAR-10 dataset: an annealing scheme for the hyperparameter α and selective knowledge distillation. Of these two, the first yields the best results. Choosing the optimal annealing scheme results in an increase in test set accuracy of 6%. Finally, we apply our method to the real-world use case of skin lesion classification. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
| Nepřihlášeným uživatelům se plný text nezobrazuje | K zobrazení výsledku je třeba se přihlásit. |