| Autor: |
Thaylon Guedes, Marta Mattoso, Marcos Bedo, Daniel de Oliveira |
| Jazyk: |
angličtina |
| Rok vydání: |
2024 |
| Předmět: |
|
| Zdroj: |
SoftwareX, Vol 28, Iss , Pp 101927- (2024) |
| Druh dokumentu: |
article |
| ISSN: |
2352-7110 |
| DOI: |
10.1016/j.softx.2024.101927 |
| Popis: |
While researchers benefit from Apache Spark for executing scientific workflows at scale, they often lack provenance support due to the framework’s design limitations. This paper presents SAMbA-RaP, a provenance extension for Apache Spark. It focuses on: (i) Executing external, black-box applications with intensive I/O operations within the workflow while leveraging Spark’s in-memory data structures, (ii) Extracting domain-specific data from in-memory data structures and (iii) Implementing data versioning and capturing the provenance graph in a workflow execution. SAMbA-RaP also provides real-time reports via a web interface, enabling scientists to explore dataflow transformations and content evolution as they run workflows. |
| Databáze: |
Directory of Open Access Journals |
| Externí odkaz: |
|
