Locality-Aware Workflow Orchestration for Big Data
| Autor: | Akif Quddus Khan, Ahmet Soylu, Mihhail Matskin, Andrei-Alin Corodescu, Nikolay Nikolov, Dumitru Roman, Amir H. Payberah |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Předmět: |
Data processing
Computer science business.industry Distributed computing Locality Big data 02 engineering and technology Workflow Software 020204 information systems 0202 electrical engineering electronic engineering information engineering Software containers 020201 artificial intelligence & image processing Data locality Orchestration (computing) Enhanced Data Rates for GSM Evolution Big data workflows business Edge computing |
| Zdroj: | MEDES Proceedings of the 13th International Conference on Management of Digital EcoSystems |
| ISSN: | 1010-1683 |
| Popis: | The development of the Edge computing paradigm shifts data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructure. Such a paradigm requires data processing solutions that consider data locality in order to reduce the performance penalties from data transfers between remote (in network terms) data centres. However, existing Big Data processing solutions have limited support for handling data locality and are inefficient in processing small and frequent events specific to Edge environments. This paper proposes a novel architecture and a proof-of-concept implementation for software container-centric Big Data workflow orchestration that puts data locality at the forefront. Our solution considers any available data locality information by default, leverages long-lived containers to execute workflow steps, and handles the interaction with different data sources through containers. We compare our system with Argo workflow and show significant performance improvements in terms of speed of execution for processing units of data using our data locality aware Big Data workflow approach. This work was partly funded by the EC H2020 project “DataCloud” (Grant nr. 101016835) and the NFR project “BigDataMine” (Grant nr. 309691). |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|