Benchmarking network fabrics for data distributed training of deep neural networks
| Autor: | Peter Michaleas, Charles Yee, Antonio Rosa, Matthew Hubbell, Siddharth Samsi, Vijay Gadepally, Albert Reuther, Julie Mullen, Andrew Prout, Bill Bergeron, Lauren Milechin, Chansup Byun, David Bestor, Anna Klein, Michael Jones, Bill Arcand, Michael Houle, Jeremy Kepner, Andrew Kirby |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Ethernet
FOS: Computer and information sciences Computer Science - Machine Learning 020203 distributed computing Computer Science - Performance SIMPLE (military communications protocol) Artificial neural network Computer science business.industry Distributed computing Deep learning Training (meteorology) 02 engineering and technology Benchmarking Machine Learning (cs.LG) Performance (cs.PF) Software Computer Science - Distributed Parallel and Cluster Computing 020204 information systems 0202 electrical engineering electronic engineering information engineering Deep neural networks Artificial intelligence Distributed Parallel and Cluster Computing (cs.DC) business |
| Zdroj: | HPEC |
| DOI: | 10.48550/arxiv.2008.08057 |
| Popis: | Artificial Intelligence/Machine Learning applications require the training of complex models on large amounts of labelled data. The large computational requirements for training deep models have necessitated the development of new methods for faster training. One such approach is the data parallel approach, where the training data is distributed across multiple compute nodes. This approach is simple to implement and supported by most of the commonly used machine learning frameworks. The data parallel approach leverages MPI for communicating gradients across all nodes. In this paper, we examine the effects of using different physical hardware interconnects and network-related software primitives for enabling data distributed deep learning. We compare the effect of using GPUDirect and NCCL on Ethernet and OmniPath fabrics. Our results show that using Ethernet-based networking in shared HPC systems does not have a significant effect on the training times for commonly used deep neural network architectures or traditional HPC applications such as Computational Fluid Dynamics. Comment: Accepted for publication at IEEE HPEC 2020 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|