MR-Advisor: A comprehensive tuning, profiling, and prediction tool for MapReduce execution frameworks on HPC clusters
| Autor: | Dhabaleswar K. Panda, Xiaoyi Lu, Dipti Shankar, Nusrat Sharmin Islam, Md. Wasi-ur Rahman |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
Profiling (computer programming)
020203 distributed computing Distributed Computing Environment Remote direct memory access Computer Networks and Communications Computer science 020206 networking & telecommunications 02 engineering and technology Parallel computing Theoretical Computer Science Artificial Intelligence Hardware and Architecture Scalability 0202 electrical engineering electronic engineering information engineering Software |
| Zdroj: | Journal of Parallel and Distributed Computing. 120:237-250 |
| ISSN: | 0743-7315 |
| Popis: | MapReduce is the most popular parallel computing framework for big data processing which allows massive scalability across distributed computing environment. Advanced RDMA-based design of Hadoop MapReduce has been proposed that alleviates the performance bottlenecks in default Hadoop MapReduce by leveraging the benefits from RDMA. On the other hand, data processing engine, Spark, provides fast execution of MapReduce applications through in-memory processing. Performance optimization for these contemporary big data processing frameworks on modern High-Performance Computing (HPC) systems is a formidable task because of the numerous configuration possibilities in each of them. In this paper, we propose MR-Advisor, a comprehensive tuning, profiling, and prediction tool for MapReduce. MR-Advisor is generalized to provide performance optimizations for Hadoop, Spark, and RDMA-enhanced Hadoop MapReduce designs over different file systems such as HDFS, Lustre, and Tachyon. Performance evaluations reveal that, with MR-Advisor’s suggested values, the job execution performance can be enhanced by a maximum of 58% over the current best-practice values for user-level configuration parameters. To the best of our knowledge, this is the first tool that supports tuning and prediction for both Apache Hadoop and Spark, as well as the RDMA and Lustre-based advanced designs. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect