PartialFormer: Modeling Part Instead of Whole for Machine Translation

Autor:	Zheng, Tong, Li, Bei, Bao, Huiwen, Wang, Jiale, Shan, Weiqiao, Xiao, Tong, Zhu, Jingbo
Rok vydání:	2023
Předmět:	Computer Science - Computation and Language Computer Science - Artificial Intelligence
Druh dokumentu:	Working Paper
Popis:	The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multi-head attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer's capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/zhengkid/PartialFormer. Comment: Accepted by ACL2024 Findings
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2310.14921 Zobrazit plný text záznamu View this record from Arxiv