Approximate Random Dropout for DNN training acceleration in GPGPU
| Autor: | Hai Zhao, Zhenghao Peng, Xiaoyao Liang, Zhuoran Song, Ru Wang, Dongyu Ru, Hongru Huang, Li Jiang |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
010302 applied physics
Speedup Artificial neural network Computer science Computation 02 engineering and technology 01 natural sciences 020202 computer hardware & architecture Synapse Search algorithm Multilayer perceptron 0103 physical sciences 0202 electrical engineering electronic engineering information engineering General-purpose computing on graphics processing units Algorithm Energy (signal processing) Dropout (neural networks) |
| Zdroj: | DATE |
| DOI: | 10.23919/date.2019.8715135 |
| Popis: | The training phases of Deep neural network (DNN) consumes enormous processing time and energy. Compression techniques utilizing the sparsity of DNNs can effectively accelerate the inference phase of DNNs. However, it can be hardly used in the training phase because the training phase involves dense matrix-multiplication using General Purpose Computation on Graphics Processors (GPGPU), which endorse regular and structural data layout. In this paper, we propose the Approximate Random Dropout that replaces the conventional random dropout of neurons and synapses with a regular and online generated patterns to eliminate the unnecessary computation and data access. We develop a SGD-based Search Algorithm that producing the distribution of dropout patterns to compensate the potential accuracy loss. We prove our approach is statistically equivalent to the previous dropout method. Experiments results on multilayer perceptron (MLP) and long short-term memory (LSTM) using well-known benchmarks show that the speedup rate brought by the proposed Approximate Random Dropout ranges from 1.18–2.16 (1.24–1.85) when dropout rate is 0.3–0.7 on MLP (LSTM) with negligible accuracy drop. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|