Toward an End-to-End Framework for Modeling, Monitoring and Anomaly Detection for Scientific Workflows
| Autor: | Vickie E. Lynch, Anirban Mandal, Rafael Ferreira da Silva, James Wynne, Rajiv Mayani, Ilya Baldin, Paul Ruth, Christopher D. Carothers, Ben Mayer, Justin M. LaPre, Gideon Juve, Ewa Deelman, Jeremy S. Meredith, Dariusz Król, Mark Blanco, Jeffrey S. Vetter, Brian Tierney |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
020203 distributed computing
Computer science Scale (chemistry) Distributed computing Anomaly (natural sciences) 020207 software engineering 02 engineering and technology Data science Data modeling Workflow Black box Scalability 0202 electrical engineering electronic engineering information engineering Anomaly detection Workflow management system |
| Zdroj: | IPDPS Workshops |
| Popis: | Modern science is often conducted on large scale, distributed, heterogeneous and high-performance computing infrastructures. Increasingly, the scale and complexity of both the applications and the underlying execution platforms have been growing. Scientific workflows have emerged as a flexible representation to declaratively express complex applications with data andcontrol dependences. However, it is extremely challengingfor scientists to execute their science workflows in a reliable and scalable way due to a lack of understanding of expected and realistic behavior of complex scientific workflows on large scale and distributed HPC systems. This is exacerbated by failures and anomalies in largescale systems and applications, which makes detecting, analyzing and acting on anomaly events challenging. In this work, we present a prototype of an end-to-end system for modeling and diagnosing the runtime performance of complex scientific workflows. We interfaced the Pegasus workflow management system, Aspen performance modeling, monitoring and anomaly detection into an integrated framework that not only improves the understanding of complex scientific applications on large scale complex infrastructure, but also detects anomalies and supports adaptivity. We present a black box modeling tool, a comprehensive online monitoring system, and anomaly detection algorithms that employ the models and monitoring data to detect anomaly events. We present an evaluation of the system with a Spallation Neutron Source workflow as a driving use case. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|