Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
| Autor: | Taiji Suzuki, Akiyuki Tanizawa, Wataru Asano, Atsushi Yaguchi, Yukinobu Sakata, Shuhei Nitta |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Channel (digital image) Machine translation Computer science Computer Vision and Pattern Recognition (cs.CV) Computer Science - Computer Vision and Pattern Recognition Machine Learning (stat.ML) 02 engineering and technology 010501 environmental sciences computer.software_genre 01 natural sciences Machine Learning (cs.LG) Reduction (complexity) Statistics - Machine Learning 0202 electrical engineering electronic engineering information engineering 0105 earth and related environmental sciences Artificial neural network business.industry Deep learning Pattern recognition 020201 artificial intelligence & image processing Node (circuits) Artificial intelligence business computer MNIST database |
| Zdroj: | ICMLA |
| DOI: | 10.1109/icmla.2018.00054 |
| Popis: | In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation. However, large DNN models are needed to achieve state-of-the-art performance, exceeding the capabilities of edge devices. Model reduction is thus needed for practical use. In this paper, we point out that deep learning automatically induces group sparsity of weights, in which all weights connected to an output channel (node) are zero, when training DNNs under the following three conditions: (1) rectified-linear-unit (ReLU) activations, (2) an $L_2$-regularized objective function, and (3) the Adam optimizer. Next, we analyze this behavior both theoretically and experimentally, and propose a simple model reduction method: eliminate the zero weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets, we demonstrate the sparsity with various training setups. Finally, we show that our method can efficiently reduce the model size and performs well relative to methods that use a sparsity-inducing regularizer. Comment: 8 pages, 7 figures, 6 tables, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA) |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|