The Vera C. Rubin Observatory Data Butler and Pipeline Execution System

Autor:	Jenness, Tim, Bosch, James F., Lust, Nate B., Pease, Nathan M., Gower, Michelle, Kowalik, Mikolaj, Dubois-Felsmann, Gregory P., Mueller, Fritz, Schellart, Pim
Rok vydání:	2022
Předmět:	Astrophysics - Instrumentation and Methods for Astrophysics Computer Science - Distributed Parallel and Cluster Computing
Druh dokumentu:	Working Paper
Popis:	The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations. Comment: 14 pages, 3 figures, submitted to Proc SPIE 12189, "Software and Cyberinfrastructure for Astronomy VII", Montreal, CA, July 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2206.14941 Zobrazit plný text záznamu View this record from Arxiv