Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers

Autor:	Sahiner, Arda, Ergen, Tolga, Ozturkler, Batu, Pauly, John, Mardani, Morteza, Pilanci, Mert
Rok vydání:	2022
Předmět:	Computer Science - Machine Learning Computer Science - Computer Vision and Pattern Recognition Mathematics - Optimization and Control
Druh dokumentu:	Working Paper
Popis:	Vision transformers using self-attention or its proposed alternatives have demonstrated promising results in many image related tasks. However, the underpinning inductive bias of attention is not well understood. To address this issue, this paper analyzes attention through the lens of convex duality. For the non-linear dot-product self-attention, and alternative mechanisms such as MLP-mixer and Fourier Neural Operator (FNO), we derive equivalent finite-dimensional convex problems that are interpretable and solvable to global optimality. The convex programs lead to {\it block nuclear-norm regularization} that promotes low rank in the latent feature and token dimensions. In particular, we show how self-attention networks implicitly clusters the tokens, based on their latent similarity. We conduct experiments for transferring a pre-trained transformer backbone for CIFAR-100 classification by fine-tuning a variety of convex attention heads. The results indicate the merits of the bias induced by attention compared with the existing MLP or linear heads. Comment: 38 pages, 2 figures. To appear in ICML 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2205.08078 Zobrazit plný text záznamu View this record from Arxiv