Spartan: A Sparsity-Adaptive Framework to Accelerate Deep Neural Network Training on GPUs
| Autor: | Nicolas Bohm Agostini, Shi Dong, José L. Abellán, David Kaeli, Yifan Sun, Elmira Karimi, Daniel Lowell, Jing Zhou, José Cano |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Profiling (computer programming)
020203 distributed computing Speedup Artificial neural network Exploit Computer science 02 engineering and technology computer.software_genre Software framework Computational Theory and Mathematics Computer engineering Hardware and Architecture Signal Processing 0202 electrical engineering electronic engineering information engineering Overhead (computing) Spartan computer Sparse matrix |
| Zdroj: | IEEE Transactions on Parallel and Distributed Systems. 32:2448-2463 |
| ISSN: | 2161-9883 1045-9219 |
| DOI: | 10.1109/tpds.2021.3067825 |
| Popis: | Deep Neural Networks (DNNs) have emerged as an important class of machine learning algorithms, providing accurate solutions to a broad range of applications. Sparsity in activation maps in DNN training presents an opportunity to reduce computations. However, exploiting activation sparsity presents two major challenges: i) profiling activation sparsity during training comes with significant overhead due to computing the degree of sparsity and the data movement; ii) the dynamic nature of activation maps requires dynamic dense-to-sparse conversion during training, leading to significant overhead. In this article, we present Spartan , a lightweight hardware/software framework to accelerate DNN training on a GPU. Spartan provides a cost-effective and programmer-transparent microarchitectural solution to exploit activation sparsity detected during training. Spartan provides an efficient sparsity monitor, a tile-based sparse GEMM algorithm, and a novel compaction engine designed for GPU workloads. Spartan can reduce sparsity profiling overhead by 52.5× on average. For the most compute-intensive layers, i.e., convolutional layers, we can speedup AlexNet by 3.4×, VGGNet-16 by 2.14×, and ResNet-18 by 2.02×, when training on the ImageNet dataset. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|