Anomaly Detection Using Program Control Flow Graph Mining From Execution Logs
| Autor: | Shubham Atreja, Subhrajit Bhattacharya, Gargi B. Dasgupta, Animesh Nandi, Atri Mandal |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
Computer science
Pipeline (computing) Volume (computing) Process (computing) 02 engineering and technology computer.software_genre 020204 information systems Scalability Spark (mathematics) 0202 electrical engineering electronic engineering information engineering Control flow graph 020201 artificial intelligence & image processing Anomaly detection Instrumentation (computer programming) Data mining computer |
| Zdroj: | KDD |
| DOI: | 10.1145/2939672.2939712 |
| Popis: | We focus on the problem of detecting anomalous run-time behavior of distributed applications from their execution logs. Specifically we mine templates and template sequences from logs to form a control flow graph (cfg) spanning distributed components. This cfg represents the baseline healthy system state and is used to flag deviations from the expected behavior of runtime logs. The novelty in our work stems from the new techniques employed to: (1) overcome the instrumentation requirements or application specific assumptions made in prior log mining approaches, (2) improve the accuracy of mined templates and the cfg in the presence of long parameters and high amount of interleaving respectively, and (3) improve by orders of magnitude the scalability of the cfg mining process in terms of volume of log data that can be processed per day. We evaluate our approach using (a) synthetic log traces and (b) multiple real-world log datasets collected at different layers of application stack. Results demonstrate that our template mining, cfg mining, and anomaly detection algorithms have high accuracy. The distributed implementation of our pipeline is highly scalable and has more than 500 GB/day of log data processing capability even on a 10 low-end VM based (Spark + Hadoop) cluster. We also demonstrate the efficacy of our end-to-end system using a case study with the Openstack VM provisioning system. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|