Big Data and HPC collocation: Using HPC idle resources for Big Data Analytics
| Autor: | Michael Mercier, David Glesser, Olivier Richard, Yiannis Georgiou |
|---|---|
| Přispěvatelé: | Laboratoire d'Informatique de Grenoble (LIG ), Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019]), Bull atos technologies, Data Aware Large Scale Computing (DATAMOVE ), Inria Grenoble - Rhône-Alpes, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Laboratoire d'Informatique de Grenoble (LIG ), Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019])-Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019]), GRID'5000 |
| Jazyk: | angličtina |
| Rok vydání: | 2017 |
| Předmět: |
Best Effort
Big Data 0209 industrial biotechnology Schedule Computer science Big data 02 engineering and technology Dynamic priority scheduling computer.software_genre Scheduling (computing) 020901 industrial engineering & automation 0202 electrical engineering electronic engineering information engineering Resource Management Resource management [INFO]Computer Science [cs] Infrastructure business.industry Scheduling Workload Supercomputer HPC and Big Data Convergence Middleware HPC Operating system 020201 artificial intelligence & image processing business computer HPDA |
| Zdroj: | IEEE BigData 2017 IEEE BigData 2017, Dec 2017, Boston, United States IEEE BigData |
| Popis: | International audience; Executing Big Data workloads upon High Performance Computing (HPC) infrastractures has become an attractive way to improve their performances. However, the collocation of HPC and Big Data workloads is not an easy task, mainly because of their core concepts' differences. This paper focuses on the challenges related to the scheduling of both Big Data and HPC workloads on the same computing platform. In classic HPC workloads, the rigidity of jobs tends to create holes in the schedule: we can use those idle resources as a dynamic pool for Big Data workloads. We propose a new idea based on Resource and Job Management System's (RJMS) configuration, that makes HPC and Big Data systems to communicate through a simple prolog/epilog mechanism. It leverages the built-in resilience of Big Data frameworks, while minimizing the disturbance on HPC workloads. We present the first study of this approach, using the production RJMS middleware OAR and Hadoop YARN from the HPC and Big Data ecosystems respectively. Our new technique is evaluated with real experiments upon the Grid5000 platform. Our experiments validate our assumptions and show promising results. The system is capable of running an HPC workload with 70% cluster utilization, with a Big Data workload that fills the schedule holes to reach a full 100% utilization. We observe a penalty on the mean waiting time for HPC jobs of less than 17% and a Big Data effectiveness of more than 68% in average. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|