Data-Centric Iteration in Dynamic Workflows

Autor: Alvaro L. G. A. Coutinho, Patrick Valduriez, Gabriel M. Guerra, Marta Mattoso, Fernando A. Rochinha, Jonas Dias
Přispěvatelé: Instituto Alberto Luiz Coimbra de Pós-Graduação e Pesquisa de Engenharia (COPPE-UFRJ), Universidade Federal do Rio de Janeiro (UFRJ), Institut de Biologie Computationnelle (IBC), Institut National de la Recherche Agronomique (INRA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Université de Montpellier (UM)-Centre National de la Recherche Scientifique (CNRS), Scientific Data Management (ZENITH), Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier (LIRMM), Centre National de la Recherche Scientifique (CNRS)-Université de Montpellier (UM)-Centre National de la Recherche Scientifique (CNRS)-Université de Montpellier (UM)-Inria Sophia Antipolis - Méditerranée (CRISAM), Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), CNPq, CAPES, FAPERJ, INRIA (MUSIC and HOSCAR projects), Inria Associated team MUSIC, CNPq-Inria HOSCAR, Université de Montpellier (UM)-Centre National de la Recherche Scientifique (CNRS)-Université de Montpellier (UM)-Centre National de la Recherche Scientifique (CNRS)-Inria Sophia Antipolis - Méditerranée (CRISAM), COPPE - Instituto Alberto Luiz Coimbra de Pós-Graduação e Pesquisa de Engenharia (COPPE-UFRJ), Universidade Federal do Rio de Janeiro [Rio de Janeiro] (UFRJ), Université de Montpellier (UM)-Institut National de la Recherche Agronomique (INRA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)
Jazyk: angličtina
Rok vydání: 2015
Předmět:
Zdroj: Future Generation Computer Systems
Future Generation Computer Systems, Elsevier, 2015, 46, pp.114-126. ⟨10.1016/j.future.2014.10.021⟩
Future Generation Computer Systems, 2015, 46, pp.114-126. ⟨10.1016/j.future.2014.10.021⟩
ISSN: 0167-739X
DOI: 10.1016/j.future.2014.10.021⟩
Popis: Dynamic workflows are scientific workflows to support computational science simulations, typically using dynamic processes based on runtime scientific data analyses. They require the ability of adapting the workflow, at runtime, based on user input and dynamic steering. Supporting data-centric iteration is an important step towards dynamic workflows because user interaction with workflows is iterative. However, current support for iteration in scientific workflows is static and does not allow for changing data at runtime. In this paper, we propose a solution based on algebraic operators and a dynamic execution model to enable workflow adaptation based on user input and dynamic steering. We introduce the concept of iteration lineage that makes provenance data management consistent with dynamic iterative workflow changes. Lineage enables scientists to interact with workflow data and configuration at runtime through an API that triggers steering. We evaluate our approach using a novel and real large-scale workflow for uncertainty quantification on a 640-core cluster. The results show impressive execution time savings from 2.5 to 24 days, compared to non-iterative workflow execution. We verify that the maximum overhead introduced by our iterative model is less than 5% of execution time. Also, our proposed steering algorithms are very efficient and run in less than 1 millisecond, in the worst-case scenario. Algebraic operators support data-centric iteration in dynamic workflows.Runtime data lineage, a concept inspired by provenance enables dynamic loops.Two algorithms support runtime adaptation of the workflow based on user input.Real-life experiment for Uncertainty Quantification in the Oil & Gas domain.A novel iterative workflow for Uncertainty Quantification is steered by users.
Databáze: OpenAIRE
Externí odkaz: