Python Workflows on HPC Systems
| Autor: | Janis Keuper, Philipp Reusch, Dominik Strasel |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning business.industry Computer science Deep learning Workaround GPU cluster Python (programming language) Data structure Machine Learning (cs.LG) CUDA Workflow Software Computer Science - Distributed Parallel and Cluster Computing Hardware_GENERAL Distributed Parallel and Cluster Computing (cs.DC) Artificial intelligence Software engineering business computer computer.programming_language |
| Zdroj: | PyHPC@SC |
| Popis: | The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user environments or parallel programming - making it quite challenging to maintain stable and secure Python workflows on a HPC system. In this paper, we analyze the key problems induced by the usage of Python on HPC clusters and sketch appropriate workarounds for efficiently maintaining multi-user Python software environments, securing and restricting resources of Python jobs and containing Python processes, while focusing on Deep Learning applications running on GPU clusters. 9 pages with 7 figures, submitted and accepted at the PyHPC Workshop at SuperComputing 2020 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|