Stocator: Providing High Performance and Fault Tolerance for Apache Spark Over Object Storage
Autor: | Elliot K. Kolodner, Effi Ofer, Pietro Michiardi, Michael Factor, Gil Vernik, Francesco Pace |
---|---|
Rok vydání: | 2018 |
Předmět: |
business.industry
Group method of data handling Computer science Semantics (computer science) Speculative execution Fault tolerance 02 engineering and technology Service provider computer.software_genre Object storage 020204 information systems Computer data storage Spark (mathematics) Data_FILES 0202 electrical engineering electronic engineering information engineering Operating system business computer |
Zdroj: | CCGrid |
DOI: | 10.1109/ccgrid.2018.00073 |
Popis: | Until now object storage has not been a first-class citizen of the Apache Hadoop ecosystem including Apache Spark. Hadoop connectors to object storage have been based on file semantics, an impedance mismatch, which leads to low performance and the need for an additional consistent storage system to achieve fault tolerance. In particular, Hadoop depends on its underlying storage system and its associated connector for fault tolerance and allowing speculative execution. However, these characteristics are obtained through file operations that are not native for object storage, and are both costly and not atomic. As a result these connectors are not efficient and more importantly they cannot help with fault tolerance for object storage. We introduce Stocator, whose novel algorithm achieves both high performance and fault tolerance by taking advantage of object storage semantics. This greatly decreases the number of operations on object storage as well as enabling a much simpler approach to dealing with the eventually consistent semantics typical of object storage. We have implemented Stocator and shared it in open source. Performance testing with Apache Spark shows that it can be 18 times faster for write intensive workloads and can perform 30 times fewer operations on object storage than the legacy Hadoop connectors, reducing costs both for the client and the object storage service provider. |
Databáze: | OpenAIRE |
Externí odkaz: |