A review on the attention mechanism of deep learning
| Autor: | Guoqiang Zhong, Zhaoyang Niu, Hui Yu |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
0209 industrial biotechnology
Network architecture business.industry Computer science Mechanism (biology) Cognitive Neuroscience Deep learning Representation (systemics) 02 engineering and technology Data science Field (computer science) Computer Science Applications Focus (linguistics) 020901 industrial engineering & automation Artificial Intelligence 0202 electrical engineering electronic engineering information engineering Feature (machine learning) 020201 artificial intelligence & image processing Artificial intelligence business Interpretability |
| Zdroj: | Neurocomputing. 452:48-62 |
| ISSN: | 0925-2312 |
| DOI: | 10.1016/j.neucom.2021.03.091 |
| Popis: | Attention has arguably become one of the most important concepts in the deep learning field. It is inspired by the biological systems of humans that tend to focus on the distinctive parts when processing large amounts of information. With the development of deep neural networks, attention mechanism has been widely used in diverse application domains. This paper aims to give an overview of the state-of-the-art attention models proposed in recent years. Toward a better general understanding of attention mechanisms, we define a unified model that is suitable for most attention structures. Each step of the attention mechanism implemented in the model is described in detail. Furthermore, we classify existing attention models according to four criteria: the softness of attention, forms of input feature, input representation, and output representation. Besides, we summarize network architectures used in conjunction with the attention mechanism and describe some typical applications of attention mechanism. Finally, we discuss the interpretability that attention brings to deep learning and present its potential future trends. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect