The Mystery of the Failing Jobs: Insights from Operational Data from Two University-Wide Computing Systems
| Autor: | Zbigniew Kalbarczyk, Rakesh Kumar, William Kramer, Saurabh Jha, Xiaohui Carol Song, Rajesh Kalyanam, Ravishankar K. Iyer, Stephen Lien Harrell, Ashraf Mahgoub, Saurabh Bagchi |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
020203 distributed computing
021110 strategic defence & security studies Downtime Computer science Quality of service Distributed computing 0211 other engineering and technologies 02 engineering and technology Supercomputer Computing systems Scheduling (computing) Proof of concept Computer cluster 0202 electrical engineering electronic engineering information engineering Data analysis |
| Zdroj: | DSN |
| Popis: | Node downtime and failed jobs in a computing cluster translate into wasted resources and user dissatisfaction. Therefore understanding why nodes and jobs fail in HPC clusters is essential. This paper provides analyses of node and job failures in two university-wide computing clusters at two Tier I US research universities. We analyzed approximately 3.0M job execution data of System A and 2.2M of System B with data sources coming from accounting logs, resource usage for all primary local and remote resources (memory, IO, network), and node failure data. We observe different kinds of correlations of failures with resource usages and propose a job failure prediction model to trigger event-driven checkpointing and avoid wasted work. Additionally, we present user history based resource usage and runtime prediction models. These models have the potential to avoid system related issues such as contention, and improve quality of service such as lower mean queue time, if their predictions are used to make a more informed scheduling decision. As a proof of concept, we simulate an easy backfill scheduler to use predictions of one of these models, i.e., runtime and show the improvements in terms of lower mean queue time. Arising out of these observations, we provide generalizable insights for cluster management to improve reliability, such as, for some execution environments local contention dominates, while for others system-wide contention dominates. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|