The Zwicky Transient Facility Alert Distribution System
| Autor: | Ben Rusholme, Matthew J. Graham, Frank J. Masci, K. Simon Krughoff, Shrinivas R. Kulkarni, George Helou, Maria T. Patterson, Mario Juric, V. Zach Golkhou, Eric C. Bellm |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
010504 meteorology & atmospheric sciences
business.industry Computer science Serialization Real-time computing FOS: Physical sciences Astronomy and Astrophysics Large Synoptic Survey Telescope Modular design Python (programming language) 01 natural sciences Stream processing Space and Planetary Science Software deployment 0103 physical sciences Scalability Use case Astrophysics - Instrumentation and Methods for Astrophysics business 010303 astronomy & astrophysics computer Instrumentation and Methods for Astrophysics (astro-ph.IM) 0105 earth and related environmental sciences computer.programming_language |
| ISSN: | 1538-3873 |
| DOI: | 10.48550/arxiv.1902.02227 |
| Popis: | The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope. Comment: Published in PASP Focus Issue on the Zwicky Transient Facility (doi: 10.1088/1538-3873/aae904). 9 Pages, 2 Figures |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|