ALOJA-ML: A Framework for Automating Characterization and Knowledge Discovery in Hadoop Deployments
| Autor: | Daron Green, Aaron Call, David Carrera, Robert L. Reinauer, Josep Ll. Berral, Nicolas Poggi |
|---|---|
| Přispěvatelé: | Universitat Politècnica de Catalunya. Departament d'Arquitectura de Computadors, Barcelona Supercomputing Center, Universitat Politècnica de Catalunya. LARCA - Laboratori d'Algorísmia Relacional, Complexitat i Aprenentatge, Universitat Politècnica de Catalunya. CAP - Grup de Computació d'Altes Prestacions |
| Jazyk: | angličtina |
| Rok vydání: | 2015 |
| Předmět: |
FOS: Computer and information sciences
Informàtica::Intel·ligència artificial::Aprenentatge automàtic [Àrees temàtiques de la UPC] Data-center management Computer science C.4 Big data Cloud computing 02 engineering and technology Machine Learning (cs.LG) Execution experiences I.2.6 Software Knowledge extraction 020204 information systems Machine learning Aprenentatge automàtic 0202 electrical engineering electronic engineering information engineering Electronic data processing -- Distributed processing business.industry Performance tuning Modeling and prediction 020206 networking & telecommunications Benchmarking Automation Computer Science - Learning Computer Science - Distributed Parallel and Cluster Computing Hadoop Software deployment Benchmark (computing) Distributed Parallel and Cluster Computing (cs.DC) business Software engineering Processament distribuït de dades |
| Zdroj: | Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining-KDD '15 Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining-KDD 15 UPCommons. Portal del coneixement obert de la UPC Universitat Politècnica de Catalunya (UPC) Recercat. Dipósit de la Recerca de Catalunya Universitat Jaume I |
| Popis: | This article presents ALOJA-Machine Learning (ALOJA-ML) an extension to the ALOJA project that uses machine learning techniques to interpret Hadoop benchmark performance data and performance tuning; here we detail the approach, efficacy of the model and initial results. Hadoop presents a complex execution environment, where costs and performance depends on a large number of software (SW) configurations and on multiple hardware (HW) deployment choices. These results are accompanied by a test bed and tools to deploy and evaluate the cost-effectiveness of the different hardware configurations, parameter tunings, and Cloud services. Despite early success within ALOJA from expert-guided benchmarking, it became clear that a genuinely comprehensive study requires automation of modeling procedures to allow a systematic analysis of large and resource-constrained search spaces. ALOJA-ML provides such an automated system allowing knowledge discovery by modeling Hadoop executions from observed benchmarks across a broad set of configuration parameters. The resulting performance models can be used to forecast execution behavior of various workloads; they allow 'a-priori' prediction of the execution times for new configurations and HW choices and they offer a route to model-based anomaly detection. In addition, these models can guide the benchmarking exploration efficiently, by automatically prioritizing candidate future benchmark tests. Insights from ALOJA-ML's models can be used to reduce the operational time on clusters, speed-up the data acquisition and knowledge discovery process, and importantly, reduce running costs. In addition to learning from the methodology presented in this work, the community can benefit in general from ALOJA data-sets, framework, and derived insights to improve the design and deployment of Big Data applications. Submitted to KDD'2015. Part of the Aloja Project. Partially funded by European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 639595) - HiEST Project |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|