Analysis and Prediction of Data Transfer Throughput for Data-Intensive Workloads
| Autor: | Erich Strohmaier, Devarshi Ghoshal, Eric Pouyoul, Kesheng Wu |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Job scheduler
020203 distributed computing Computer science business.industry Distributed computing Big data 020206 networking & telecommunications 02 engineering and technology Network monitoring computer.software_genre Supercomputer 0202 electrical engineering electronic engineering information engineering Performance prediction Resource management Heuristics business Throughput (business) computer Host (network) |
| Zdroj: | IEEE BigData |
| DOI: | 10.1109/bigdata47090.2019.9005543 |
| Popis: | Scientific workflows are increasingly transferring large amounts of data between high performance computing (HPC) systems. Even though these HPC systems are connected via high-speed dedicated networks and use dedicated data transfer nodes (DTNs), it is still difficult to predict the data transfer throughput because of variations in data transfer protocols, host configurations, performance of file systems, and overlapping workloads. In order to provide reliable performance prediction for better resource management and job scheduling, we need models for predicting data transfer throughput under real-world conditions. In this paper, we explore different machine learning approaches for building data-driven models to improve performance and prediction of large-scale data transfer throughput. In addition to the variables already collected by the network monitoring system, we also develop heuristics to derive additional metrics for improving the prediction accuracy. We use the prediction results to identify the importance of different network parameters in predicting the throughput for large-scale data transfers. Through extensive tests, we identify key network parameters, discover interesting variations among different HPC sites, and show that we can predict throughput with high accuracy. We also analyze our models and results to provide recommendations for improving the performance of big data transfers. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|