Importance Estimation for Neural Network Pruning
| Autor: | Pavlo Molchanov, Jan Kautz, Arun Mallya, Iuri Frosio, Stephen Tyree |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Artificial neural network Computer science business.industry Computer Vision and Pattern Recognition (cs.CV) Deep learning Computer Science - Computer Vision and Pattern Recognition Machine Learning (stat.ML) 020206 networking & telecommunications 02 engineering and technology Filter (signal processing) Network layer Machine Learning (cs.LG) Reduction (complexity) Statistics - Machine Learning 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Sensitivity (control systems) Artificial intelligence Pruning (decision trees) business Algorithm |
| Zdroj: | CVPR |
| DOI: | 10.1109/cvpr.2019.01152 |
| Popis: | Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|