Improving Disentangled Text Representation Learning with Information-Theoretic Guidance
Autor: | Christopher Malon, Martin Renqiang Min, Pengyu Cheng, Yitong Li, Dinghan Shen, Yizhe Zhang, Lawrence Carin |
---|---|
Rok vydání: | 2020 |
Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Computer science business.industry Machine Learning (stat.ML) 02 engineering and technology Mutual information 010501 environmental sciences Information theory computer.software_genre Semantics 01 natural sciences Measure (mathematics) Machine Learning (cs.LG) Statistics - Machine Learning 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Artificial intelligence Representation (mathematics) business Feature learning computer Natural language Natural language processing 0105 earth and related environmental sciences |
Zdroj: | ACL |
Popis: | Learning disentangled representations of natural language is essential for many NLP tasks, e.g., conditional text generation, style transfer, personalized dialogue systems, etc. Similar problems have been studied extensively for other forms of data, such as images and videos. However, the discrete nature of natural language makes the disentangling of textual representations more challenging (e.g., the manipulation over the data space cannot be easily achieved). Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text, without any supervision on semantics. A new mutual information upper bound is derived and leveraged to measure dependence between style and content. By minimizing this upper bound, the proposed method induces style and content embeddings into two independent low-dimensional spaces. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation in terms of content and style preservation. Comment: Accepted by the 58th Annual Meeting of the Association for Computational Linguistics (ACL2020) |
Databáze: | OpenAIRE |
Externí odkaz: |