A convergence of key-value storage systems from clouds to supercomputers
| Autor: | Dongfang Zhao, Tonglin Li, Xiaobing Zhou, Ioan Raicu, Zhao Zhang, Iman Sadooghi, Ke Wang |
|---|---|
| Rok vydání: | 2015 |
| Předmět: |
020203 distributed computing
Computer Networks and Communications business.industry Computer science Distributed computing 020206 networking & telecommunications Fault tolerance Cloud computing 02 engineering and technology computer.software_genre Supercomputer Computer Science Applications Theoretical Computer Science Software portability Computational Theory and Mathematics Computer cluster Computer data storage Scalability 0202 electrical engineering electronic engineering information engineering Operating system business Distributed File System computer Message queue Software |
| Zdroj: | Concurrency and Computation: Practice and Experience. 28:44-69 |
| ISSN: | 1532-0626 |
| DOI: | 10.1002/cpe.3614 |
| Popis: | This paper presents a convergence of distributed key-value storage systems in clouds and supercomputers. It specifically presents ZHT, a zero-hop distributed key-value store system, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. ZHT has some important properties, such as being lightweight, dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append, compare and swap, callback in addition to the traditional insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 64 nodes, an Amazon EC2 virtual cluster up to 96 nodes, to an IBM Blue Gene/P supercomputer with 8K nodes. We compared ZHT against other key-value stores and found it offers superior performance for the features and portability it supports. This paper also presents several real systems that have adopted ZHT, namely, FusionFS a distributed file system, IStore a storage system with erasure coding, MATRIX distributed scheduling, Slurm++ distributed HPC job launch, Fabriq distributed message queue management; all of these real systems have been simplified because of key-value storage systems and have been shown to outperform other leading systems by orders of magnitude in some cases. It is important to highlight that some of these systems are rooted in HPC systems from supercomputers, while others are rooted in clouds and ad hoc distributed systems; through our work, we have shown how versatile key-value storage systems can be in such a variety of environments. Copyright © 2015 John Wiley & Sons, Ltd. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Plný text