On the importance of sluggish state memory for learning long term dependency
| Autor: | H. M. Powell, Mahmud S. Shertil, Jonathan Tepper |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2016 |
| Předmět: |
Vanishing gradient problem
Information Systems and Management Dependency (UML) Computer science business.industry 02 engineering and technology Grammar induction Management Information Systems Term (time) Task (project management) 03 medical and health sciences Variable (computer science) 0302 clinical medicine Artificial Intelligence 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing State (computer science) Artificial intelligence Echo state network business 030217 neurology & neurosurgery Software |
| ISSN: | 0950-7051 |
| Popis: | The vanishing gradients problem inherent in Simple Recurrent Networks (SRN) trained with back-propagation, has led to a significant shift towards the use of Long Short-term Memory (LSTM) and Echo State Networks (ESN), which overcome this problem through either second order error-carousel schemes or different learning algorithms respectively. This paper re-opens the case for SRN-based approaches, by considering a variant, the Multi-recurrent Network (MRN). We show that memory units embedded within its architecture can ameliorate against the vanishing gradient problem, by providing variable sensitivity to recent and more historic information through layer- and self-recurrent links with varied weights, to form a so-called sluggish state-based memory. We demonstrate that an MRN, optimised with noise injection, is able to learn the long term dependency within a complex grammar induction task, significantly outperforming the SRN, NARX and ESN. Analysis of the internal representations of the networks, reveals that sluggish state-based representations of the MRN are best able to latch on to critical temporal dependencies spanning variable time delays, to maintain distinct and stable representations of all underlying grammar states. Surprisingly, the ESN was unable to fully learn the dependency problem, suggesting the major shift towards this class of models may be premature. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect