Match memory recurrent networks
Autor: | Tom Vodopivec, Spyridon Samothrakis, Maria Fasli, Michael Fairbank |
---|---|
Rok vydání: | 2016 |
Předmět: |
Artificial neural network
Computer science business.industry Deep learning 05 social sciences 010501 environmental sciences 01 natural sciences Recurrent neural network medicine.anatomical_structure 0502 economics and business Softmax function Question answering medicine Artificial intelligence Neuron 050207 economics Types of artificial neural networks business 0105 earth and related environmental sciences |
Zdroj: | IJCNN |
Popis: | Imbuing neural networks with memory and attention mechanisms allows for better generalisation with fewer data samples. By focusing only on the relevant parts of data, which is encoded in an internal “memory” format, the network is able to infer better and more reliable patterns. Most neuronal attention mechanisms are based on internal networks structures that impose a similarity metric (e.g., dot-product), followed by some (soft-)max operator. In this paper, we propose a novel attention method based on a function between neuron activities, which we term a “match function”, which is augmented by a recursive softmax function. We evaluate the algorithm on the bAbI question answering dataset and show that it has stronger performance when only one memory hop is used in both terms of average score and in terms the number of solved questions. Furthermore, with three memory hops, our algorithm can solve 12/20 benchmark questions using 1000 training samples per task. This is an improvement on the previous state of the art of 9/20 solved questions, which was held by end-to-end memory networks. |
Databáze: | OpenAIRE |
Externí odkaz: |