Learning Compressed Sentence Representations for On-Device Text Processing
Autor: | Asli Celikyilmaz, Lawrence Carin, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Dinghan Shen, Meng Tang, Pengyu Cheng |
---|---|
Rok vydání: | 2019 |
Předmět: |
Text corpus
FOS: Computer and information sciences Computer Science - Machine Learning Computer Science - Computation and Language Computer science business.industry 02 engineering and technology 010501 environmental sciences computer.software_genre 01 natural sciences Machine Learning (cs.LG) Semantic similarity Text processing 0202 electrical engineering electronic engineering information engineering ComputingMethodologies_DOCUMENTANDTEXTPROCESSING 020201 artificial intelligence & image processing Artificial intelligence business computer Computation and Language (cs.CL) Sentence Natural language processing 0105 earth and related environmental sciences |
Zdroj: | Scopus-Elsevier ACL (1) |
DOI: | 10.48550/arxiv.1906.08340 |
Popis: | Vector representations of sentences, trained on massive text corpora, are widely used as generic sentence embeddings across a variety of NLP problems. The learned representations are generally assumed to be continuous and real-valued, giving rise to a large memory footprint and slow retrieval speed, which hinders their applicability to low-resource (memory and computation) platforms, such as mobile devices. In this paper, we propose four different strategies to transform continuous and generic sentence embeddings into a binarized form, while preserving their rich semantic information. The introduced methods are evaluated across a wide range of downstream tasks, where the binarized sentence embeddings are demonstrated to degrade performance by only about 2% relative to their continuous counterparts, while reducing the storage requirement by over 98%. Moreover, with the learned binary representations, the semantic relatedness of two sentences can be evaluated by simply calculating their Hamming distance, which is more computational efficient compared with the inner product operation between continuous embeddings. Detailed analysis and case study further validate the effectiveness of proposed methods. Comment: To appear at ACL 2019 |
Databáze: | OpenAIRE |
Externí odkaz: |