Gated Echo State Networks: a preliminary study
| Autor: | Alessio Micheli, Claudio Gallicchio, Daniele Di Sarli |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
business.industry
Computer science Echo (computing) Reservoir computing Context (language use) 02 engineering and technology Gating Echo State Networks Gated Recurrent Neural Networks Machine learning computer.software_genre Backpropagation 03 medical and health sciences 0302 clinical medicine Recurrent neural network 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing State (computer science) Artificial intelligence business Gradient descent computer 030217 neurology & neurosurgery |
| Zdroj: | INISTA 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA) |
| DOI: | 10.1109/inista49547.2020.9194681 |
| Popis: | Gating mechanisms are widely used in the context of Recurrent Neural Networks (RNNs) to improve the network's ability to deal with long-term dependencies within the data. The typical approach for training such networks involves the expensive algorithm of gradient descent and backpropagation. On the other hand, Reservoir Computing (RC) approaches like Echo State Networks (ESNs) are extremely efficient in terms of training time and resources thanks to their use of randomly initialized parameters that do not need to be trained. Unfortunately, basic ESNs are also unable to effectively deal with complex long-term dependencies. In this work, we start investigating the problem of equipping ESNs with gating mechanisms. Under rigorous experimental settings, we compare the behaviour of an ESN with randomized gate parameters (initialized with RC techniques) against several other models, among which a leaky ESN and a fully trained gated RNN. We observe that the use of randomized gates by itself can increase the predictive accuracy of a ESN, but this increase is not meaningful when compared with other techniques. Given these results, we propose a research direction for successfully designing ESN models with gating mechanisms. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|