Exploring Erasure Coding Techniques for High Availability of Intermediate Data
| Autor: | Zhe Zhang, Derek Weitzel, David Swanson, Brian Bockelman |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Computer science Distributed computing 020206 networking & telecommunications 02 engineering and technology Storage efficiency Replication (computing) Computer Science - Distributed Parallel and Cluster Computing Data redundancy 020204 information systems High availability Metric (mathematics) 0202 electrical engineering electronic engineering information engineering Bandwidth (computing) Redundancy (engineering) Distributed Parallel and Cluster Computing (cs.DC) Erasure code |
| Zdroj: | CCGRID |
| Popis: | Scientific computing workflows generate enormous distributed data that is short-lived, yet critical for job completion time. This class of data is called intermediate data. A common way to achieve high data availability is to replicate data. However, an increasing scale of intermediate data generated in modern scientific applications demands new storage techniques to improve storage efficiency. Erasure Codes, as an alternative, can use less storage space while maintaining similar data availability. In this paper, we adopt erasure codes for storing intermediate data and compare its performance with replication. We also use the metric of Mean-Time-To-Data-Loss (MTTDL) to estimate the lifetime of intermediate data. We propose an algorithm to proactively relocate data redundancy from vulnerable machines to reliable ones to improve data availability with some extra network overhead. Furthermore, we propose an algorithm to assign redundancy units of data physically close to each other on the network to reduce the network bandwidth for reconstructing data when it is being accessed. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|