A Bayesian Optimization Framework for Neural Network Compression
| Autor: | Jacques Cali, Maxim Berman, Xingchen Ma, Christos Sagonas, Amal Rannen Triki, Matthew B. Blaschko |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Artificial neural network
Computer science Generalization business.industry Gaussian Bayesian optimization Probabilistic logic 02 engineering and technology 010501 environmental sciences 01 natural sciences Set (abstract data type) symbols.namesake Discriminative model Compression (functional analysis) 0202 electrical engineering electronic engineering information engineering symbols 020201 artificial intelligence & image processing Artificial intelligence business Algorithm 0105 earth and related environmental sciences |
| Zdroj: | ICCV |
| DOI: | 10.1109/iccv.2019.01037 |
| Popis: | Neural network compression is an important step for deploying neural networks where speed is of high importance, or on devices with limited memory. It is necessary to tune compression parameters in order to achieve the desired trade-off between size and performance. This is often done by optimizing the loss on a validation set of data, which should be large enough to approximate the true risk and therefore yield sufficient generalization ability. However, using a full validation set can be computationally expensive. In this work, we develop a general Bayesian optimization framework for optimizing functions that are computed based on U-statistics. We propagate Gaussian uncertainties from the statistics through the Bayesian optimization framework yielding a method that gives a probabilistic approximation certificate of the result. We then apply this to parameter selection in neural network compression. Compression objectives that can be written as U-statistics are typically based on empirical risk and knowledge distillation for deep discriminative models. We demonstrate our method on VGG and ResNet models, and the resulting system can find optimal compression parameters for relatively high-dimensional parametrizations in a matter of minutes on a standard desk- top machine, orders of magnitude faster than competing methods. ispartof: pages:10273-10282 ispartof: Proceedings of the IEEE International Conference on Computer Vision pages:10273-10282 ispartof: International conference on computer vision - ICCV 2019 location:Seoul, Korea date:27 Oct - 2 Nov 2019 status: published |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|