Gradient-Based Learning of Compositional Dynamics with Modular RNNs
| Autor: | Patricia Rubisch, Sebastian Otte, Martin V. Butz |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Forcing (recursion theory)
business.industry Computer science Noise (signal processing) SIGNAL (programming language) Reservoir computing 02 engineering and technology Modular design Machine learning computer.software_genre 01 natural sciences Task (computing) Recurrent neural network Simple (abstract algebra) 0103 physical sciences 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Artificial intelligence 010306 general physics business computer |
| Zdroj: | Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation ISBN: 9783030304867 ICANN (1) |
| DOI: | 10.1007/978-3-030-30487-4_38 |
| Popis: | Learning compositional dynamics with recurrent neural networks (RNNs) trained with back-propagation through time (BPTT) is usually a difficult task. Typically RNNs learn the consecutive shape along target sequences from time step to time step, focusing on local temporal correlations. When the challenge is to identify and model independent, unknown data subcomponents, that is, data generating causes on-the-fly during training, however, this local temporal shape-oriented inductive learning bias is obstructive. We propose a modular, compositional RNN architecture and derive simple procedures to automatically infer the source subdynamics that generate the data. We show that the involved error signal separation can be used for both teacher forcing and model-distinct target signal provision in the compositional RNN architecture. As a result, the entire network is able to learn compositional dynamics, developing emergent, flexibly adaptable signal decompositions within the distributed modules. We demonstrate that in this way simple RNNs trained with BPTT can learn sequences that could so far only be solved effectively with reservoir computing approaches. Moreover we show that these RNNs are much more robust against signal noise when compared to traditional BPTT or reservoir computing approaches. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|