Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression

Autor:	Wu, Feijie, He, Shiqi, Guo, Song, Qu, Zhihao, Wang, Haozhao, Zhuang, Weihua, Zhang, Jie
Rok vydání:	2022
Předmět:	Computer Science - Machine Learning Computer Science - Distributed Parallel and Cluster Computing
Druh dokumentu:	Working Paper
DOI:	10.1145/3489517.3530417
Popis:	Traditional one-bit compressed stochastic gradient descent can not be directly employed in multi-hop all-reduce, a widely adopted distributed training paradigm in network-intensive high-performance computing systems such as public clouds. According to our theoretical findings, due to the cascading compression, the training process has considerable deterioration on the convergence performance. To overcome this limitation, we implement a sign-bit compression-based learning synchronization framework, Marsit. It prevents cascading compression via an elaborate bit-wise operation for unbiased sign aggregation and its specific global compensation mechanism for mitigating compression deviation. The proposed framework retains the same theoretical convergence rate as non-compression mechanisms. Experimental results demonstrate that Marsit reduces up to 35% training time while preserving the same accuracy as training without compression. Comment: Accepted by DAC 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2204.06787 Zobrazit plný text záznamu View this record from Arxiv