Exploring stream parallel patterns in distributed MPI environments
| Autor: | Javier Muñoz, J. Daniel Garcia, David del Rio Astorga, Manuel F. Dolz, Javier López-Gómez |
|---|---|
| Přispěvatelé: | European Commission, Ministerio de Economía y Competitividad (España) |
| Rok vydání: | 2019 |
| Předmět: |
Informática
Computer Networks and Communications Computer science Distributed computing Generic programming 010103 numerical & computational mathematics 01 natural sciences Computer Graphics and Computer-Aided Design C++ programming Theoretical Computer Science Stream processing 010101 applied mathematics Artificial Intelligence Hardware and Architecture Leverage (statistics) Parallel patterns 0101 mathematics Distributed patterns Queue Software |
| Zdroj: | e-Archivo: Repositorio Institucional de la Universidad Carlos III de Madrid Universidad Carlos III de Madrid (UC3M) e-Archivo. Repositorio Institucional de la Universidad Carlos III de Madrid instname Parallel Computing |
| ISSN: | 0167-8191 |
| DOI: | 10.1016/j.parco.2019.03.004 |
| Popis: | In recent years, the large volumes of stream data and the near real-time requirements of data streaming applications have exacerbated the need for new scalable algorithms and programming interfaces for distributed and shared-memory platforms. To contribute in this direction, this paper presents a new distributed MPI back end for GrPPI, a C++ high-level generic interface of data-intensive and stream processing parallel patterns. This back end, as a new execution policy, supports distributed and hybrid (distributed+shared-memory) parallel executions of the Pipeline and Farm patterns, where the hybrid mode combines the MPI policy with a GrPPI shared-memory one. These patterns internally leverage distributed queues, which can be configured to use two-sided or one-sided MPI primitives to communicate items among nodes. A detailed analysis of the GrPPI MPI execution policy reports considerable benefits from the programmability, flexibility and readability points of view. The experimental evaluation of two different streaming applications with different distributed and shared-memory scenarios reports considerable performance gains with respect to the sequential versions at the expense of negligible GrPPI overheads. This work was partially supported by the EU project No. 801091 "ASPIDE: Exascale programming models for extreme data process ing"; and the project TIN2013-41350-P "Scalable Data Management Techniques for High-End Computing Systems" from the Ministerio de Economía y Competitividad , Spain. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect