An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collectives
| Autor: | Greg Thorson, Benjamin Klenk, Larry R. Dennison, Nan Jiang |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
010302 applied physics
Network architecture Artificial neural network Multicast Computer science business.industry Distributed computing Parallel algorithm 02 engineering and technology Network Endpoint 01 natural sciences 020202 computer hardware & architecture Software 0103 physical sciences Scalability 0202 electrical engineering electronic engineering information engineering Hardware acceleration business |
| Zdroj: | ISCA |
| Popis: | The slowdown of single-chip performance scaling combined with the growing demands of computing ever larger problems efficiently has led to a renewed interest in distributed architectures and specialized hardware. Dedicated accelerators for common or critical operations are becoming cost-effective additions to processors, peripherals, and networks. In this paper we focus on one such operation, the All-Reduce, which is both a common and critical feature of neural network training. All-Reduce is impossible to fully parallelize and difficult to amortize, so it benefits greatly from hardware acceleration. We are proposing an accelerator-centric, shared-memory network that improves All-Reduce performance through in-network reductions, as well as accelerating other collectives like Multicast. We propose switch designs to support in-network computation, including two reduction methods that offer trade-offs in implementation complexity and performance. Additionally, we propose network endpoint modifications to further improve collectives. We present simulation results for a 16 GPU system showing that our collective acceleration design improves the All-Reduce operation by up to 2x for large messages and up to 18x for small messages when compared with a state-of-the-art software algorithm, leading up to 1.4x faster DL training times for networks like Transformer. We demonstrate that this design is scalable to large systems and present results for up to 128 GPUs. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|