Neural Networks Compression for Language Modeling

Autor:	Grachev, Artem M., Ignatov, Dmitry I., Savchenko, Andrey V.
Rok vydání:	2017
Předmět:	Statistics - Machine Learning Computer Science - Computation and Language Computer Science - Machine Learning Computer Science - Neural and Evolutionary Computing 62M45 68T50 I.2.7 I.2.6 I.5.1 I.5.4
Druh dokumentu:	Working Paper
DOI:	10.1007/978-3-319-69900-4_44
Popis:	In this paper, we consider several compression techniques for the language modeling problem based on recurrent neural networks (RNNs). It is known that conventional RNNs, e.g, LSTM-based networks in language modeling, are characterized with either high space complexity or substantial inference time. This problem is especially crucial for mobile applications, in which the constant interaction with the remote server is inappropriate. By using the Penn Treebank (PTB) dataset we compare pruning, quantization, low-rank factorization, tensor train decomposition for LSTM networks in terms of model size and suitability for fast inference. Comment: Keywords: LSTM, RNN, language modeling, low-rank factorization, pruning, quantization. Published by Springer in the LNCS series, 7th International Conference on Pattern Recognition and Machine Intelligence, 2017
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1708.05963 Zobrazit plný text záznamu View this record from Arxiv