In-Memory Indexed Caching for Distributed Data Processing
| Autor: | Uta, Alexandru, Ghit, Bogdan, Dave, Ankur, Rellermeyer, Jan S., Boncz, Peter, O'Conner, L. |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2022 |
| Předmět: | |
| Zdroj: | Proceedings of the 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS) |
| Popis: | Powerful abstractions such as dataframes are only as efficient as their underlying runtime system. The de-facto distributed data processing framework, Apache Spark, is poorly suited for the modern cloud-based data-science workloads due to its outdated assumptions: static datasets analyzed using coarse-grained transformations. In this paper, we introduce the Indexed DataFrame, an in-memory cache that supports a dataframe abstraction which incorporates indexing capabilities to support fast lookup and join operations. Moreover, it supports appends with multi-version concurrency control. We implement the Indexed DataFrame as a lightweight, standalone library which can be integrated with minimum effort in existing Spark programs. We analyze the performance of the Indexed DataFrame in cluster and cloud deployments with real-world datasets and benchmarks using both Apache Spark and Databricks Runtime. In our evaluation, we show that the Indexed DataFrame significantly speeds-up query execution when compared to a non-indexed dataframe, incurring modest memory overhead. Accepted for publication at IEEE IPDPS 2022 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|