SparseTrain
| Autor: | Christopher W. Fletcher, Christopher J. Hughes, Josep Torrellas, Zhangxiaowen Gong, Houxiang Ji |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
010302 applied physics
Computer science business.industry Computation Deep learning Inference 02 engineering and technology External Data Representation 01 natural sciences 020202 computer hardware & architecture Convolution Feature (computer vision) 0103 physical sciences 0202 electrical engineering electronic engineering information engineering Leverage (statistics) SIMD Artificial intelligence business Algorithm |
| Zdroj: | PACT |
| DOI: | 10.1145/3410463.3414655 |
| Popis: | Our community has improved the efficiency of deep learning applications by exploiting sparsity in inputs. Most of that work, though, is for inference, where weight sparsity is known statically, and/or for specialized hardware. In this paper, we propose SparseTrain, a software-only scheme to leverage dynamic sparsity during training on general-purpose SIMD processors. SparseTrain exploits zeros introduced by the ReLU activation function to both feature maps and their gradients. Exploiting such sparsity is challenging because the sparsity degree is moderate and the locations of zeros change over time. SparseTrain identifies zeros in a dense data representation and performs vectorized computation. Variations of the scheme are applicable to all major components of training: forward propagation, backward propagation by inputs, and backward propagation by weights. Our experiments on a 6-core Intel Skylake-X server show that SparseTrain is very effective. In end-to-end training of VGG16, ResNet-34, and ResNet-50 with ImageNet, SparseTrain outperforms a highly-optimized direct convolution on the non-initial convolutional layers by 2.19x, 1.37x, and 1.31x, respectively. SparseTrain also benefits inference. It accelerates the non-initial convolutional layers of the aforementioned models by 1.88x, 1.64x, and 1.44x, respectively. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|