Implementation of Lightweight Convolutional Neural Networks via Layer-Wise Differentiable Compression
| Autor: | Shaoyun Xu, Gongyan Li, Huabin Diao, Yuexing Hao |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Předmět: |
Computer science
Computation TP1-1185 02 engineering and technology layer-wise differentiable Biochemistry Convolutional neural network Article Analytical Chemistry Compression (functional analysis) convolutional neural networks 0202 electrical engineering electronic engineering information engineering Pruning (decision trees) Differentiable function Electrical and Electronic Engineering Instrumentation Chemical technology structural compression Data compression ratio Atomic and Molecular Physics and Optics 020202 computer hardware & architecture 020201 artificial intelligence & image processing Performance improvement Gradient descent Algorithm |
| Zdroj: | Sensors Volume 21 Issue 10 Sensors (Basel, Switzerland) Sensors, Vol 21, Iss 3464, p 3464 (2021) |
| ISSN: | 1424-8220 |
| DOI: | 10.3390/s21103464 |
| Popis: | Convolutional neural networks (CNNs) have achieved significant breakthroughs in various domains, such as natural language processing (NLP), and computer vision. However, performance improvement is often accompanied by large model size and computation costs, which make it not suitable for resource-constrained devices. Consequently, there is an urgent need to compress CNNs, so as to reduce model size and computation costs. This paper proposes a layer-wise differentiable compression (LWDC) algorithm for compressing CNNs structurally. A differentiable selection operator OS is embedded in the model to compress and train the model simultaneously by gradient descent in one go. Instead of pruning parameters from redundant operators by contrast to most of the existing methods, our method replaces the original bulky operators with more lightweight ones directly, which only needs to specify the set of lightweight operators and the regularization factor in advance, rather than the compression rate for each layer. The compressed model produced by our method is generic and does not need any special hardware/software support. Experimental results on CIFAR-10, CIFAR-100 and ImageNet have demonstrated the effectiveness of our method. LWDC obtains more significant compression than state-of-the-art methods in most cases, while having lower performance degradation. The impact of lightweight operators and regularization factor on the compression rate and accuracy also is evaluated. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
| Nepřihlášeným uživatelům se plný text nezobrazuje | K zobrazení výsledku je třeba se přihlásit. |