Exploiting Apache Spark platform for CMS computing analytics
Autor: | Justinas Rumševičius, Daniele Bonacorsi, T. Boccali, Luca Menichetti, Valentin Kuznetsov, Marco Meoni |
---|---|
Jazyk: | angličtina |
Rok vydání: | 2017 |
Předmět: |
History
Computer science Big data Other Fields of Physics FOS: Physical sciences 02 engineering and technology computer.software_genre 01 natural sciences Education physics.data-an 0103 physical sciences Spark (mathematics) CERN CMS HEP Big Data Hadoop Spark analytics machine learning 010306 general physics Large Hadron Collider business.industry Probability and statistics Computational Physics (physics.comp-ph) 021001 nanoscience & nanotechnology Computer Science Applications Analytics physics.comp-ph Physics - Data Analysis Statistics and Probability Operating system 0210 nano-technology business computer Physics - Computational Physics Data Analysis Statistics and Probability (physics.data-an) |
Zdroj: | 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT2017) 21–25 August 2017, Seattle, United States of America, Bristol : IOP Publishing Ltd., 2018, art. no. 032055, p. 1-9 |
Popis: | The CERN IT provides a set of Hadoop clusters featuring more than 5 PBytes of raw storage with different open-source, user-level tools available for analytical purposes. The CMS experiment started collecting a large set of computing meta-data, e.g. dataset, file access logs, since 2015. These records represent a valuable, yet scarcely investigated, set of information that needs to be cleaned, categorized and analyzed. CMS can use this information to discover useful patterns and enhance the overall efficiency of the distributed data, improve CPU and site utilization as well as tasks completion time. Here we present evaluation of Apache Spark platform for CMS needs. We discuss two main use-cases CMS analytics and ML studies where efficient process billions of records stored on HDFS plays an important role. We demonstrate that both Scala and Python (PySpark) APIs can be successfully used to execute extremely I/O intensive queries and provide valuable data insight from collected meta-data. Submitted to ACAT 2017 conference proceedings |
Databáze: | OpenAIRE |
Externí odkaz: |