Compositional generalization in a deep seq2seq model by separating syntax and semantics

Autor:	Russin, Jake, Jo, Jason, O'Reilly, Randall C., Bengio, Yoshua
Rok vydání:	2019
Předmět:	Computer Science - Machine Learning Computer Science - Computation and Language Statistics - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in neuroscience suggesting separate brain systems for syntactic and semantic processing, we implement a modification to standard approaches in neural machine translation, imposing an analogous separation. The novel model, which we call Syntactic Attention, substantially outperforms standard methods in deep learning on the SCAN dataset, a compositional generalization task, without any hand-engineered features or additional supervision. Our work suggests that separating syntactic from semantic learning may be a useful heuristic for capturing compositional structure. Comment: 18 pages, 15 figures, preprint version of submission to NeurIPS 2019, under review
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1904.09708 Zobrazit plný text záznamu View this record from Arxiv