Less is More: Pay Less Attention in Vision Transformers
| Autor: | Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, Jianfei Cai |
|---|---|
| Rok vydání: | 2021 |
| Předmět: | |
| DOI: | 10.48550/arxiv.2105.14217 |
| Popis: | Transformers have become one of the dominant architectures in deep learning, particularly as a powerful alternative to convolutional neural networks (CNNs) in computer vision. However, Transformer training and inference in previous works can be prohibitively expensive due to the quadratic complexity of self-attention over a long sequence of representations, especially for high-resolution dense prediction tasks. To this end, we present a novel Less attention vIsion Transformer (LIT), building upon the fact that the early self-attention layers in Transformers still focus on local patterns and bring minor benefits in recent hierarchical vision Transformers. Specifically, we propose a hierarchical Transformer where we use pure multi-layer perceptrons (MLPs) to encode rich local patterns in the early stages while applying self-attention modules to capture longer dependencies in deeper layers. Moreover, we further propose a learned deformable token merging module to adaptively fuse informative patches in a non-uniform manner. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation, serving as a strong backbone for many vision tasks. Code is available at: https://github.com/zhuang-group/LIT Comment: Accepted to AAAI 2022 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|