On Causal Inference for Data-free Structured Pruning

Autor:	Ferianc, Martin, Sankaran, Anush, Mastropietro, Olivier, Saboori, Ehsan, Cappart, Quentin
Rok vydání:	2021
Předmět:	Computer Science - Machine Learning Statistics - Methodology
Druh dokumentu:	Working Paper
Popis:	Neural networks (NNs) are making a large impact both on research and industry. Nevertheless, as NNs' accuracy increases, it is followed by an expansion in their size, required number of compute operations and energy consumption. Increase in resource consumption results in NNs' reduced adoption rate and real-world deployment impracticality. Therefore, NNs need to be compressed to make them available to a wider audience and at the same time decrease their runtime costs. In this work, we approach this challenge from a causal inference perspective, and we propose a scoring mechanism to facilitate structured pruning of NNs. The approach is based on measuring mutual information under a maximum entropy perturbation, sequentially propagated through the NN. We demonstrate the method's performance on two datasets and various NNs' sizes, and we show that our approach achieves competitive performance under challenging conditions. Comment: Accepted to ITCI'22: The AAAI-22 Workshop on Information-Theoretic Methods for Causal Inference and Discovery
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2112.10229 Zobrazit plný text záznamu View this record from Arxiv