MemEFS: A network-aware elastic in-memory runtime distributed file system
| Autor: | Stefania Costache, Andreea Sandu, Thilo Kielmann, Ana-Maria Oprescu, Alexandru Uta, Ove Danner, Cas van der Weegen |
|---|---|
| Přispěvatelé: | Computer Systems, Network Institute, High Performance Distributed Computing |
| Jazyk: | angličtina |
| Rok vydání: | 2018 |
| Předmět: |
Speedup
Computer Networks and Communications Computer science Distributed computing 02 engineering and technology Big data and HPC systems Network variability Software Scalable computing Network adaptation 0202 electrical engineering electronic engineering information engineering Bandwidth (computing) Distributed File System High-performance I/O Large-scale scientific computing business.industry Large-scale systems for computational sciences 020206 networking & telecommunications 020207 software engineering Elasticity Workflow In-memory file system Hardware and Architecture Parallelism (grammar) Distributed hashing business Big data for e-Science |
| Zdroj: | Future Generation Computer Systems, 82(May), 631-646. Elsevier Uta, A, Danner, O, van der Weegen, C, Oprescu, A M, Sandu, A, Costache, S & Kielmann, T 2018, ' MemEFS: A network-aware elastic in-memory runtime distributed file system ', Future Generation Computer Systems, vol. 82, no. May, pp. 631-646 . https://doi.org/10.1016/j.future.2017.03.017 |
| ISSN: | 0167-739X |
| DOI: | 10.1016/j.future.2017.03.017 |
| Popis: | Scientific domains such as astronomy or bioinformatics produce increasingly large amounts of data that need to be analyzed. Such analyses are modeled as scientific workflows — applications composed of many individual tasks that exhibit data dependencies. Typically, these applications suffer from significant variability in the interplay between achieved parallelism and data footprint. To efficiently tackle the data deluge, cost effective solutions need to be deployed by extending private computing infrastructures with public cloud resources. To achieve this, two key features for such systems need to be addressed: elasticity and network adaptability. The former improves compute resource utilization efficiency, while the latter improves network utilization efficiency, since public clouds suffer from significant bandwidth variability. This paper extends our previous work on MemEFS, an in-memory elastic distributed file system by adding network adaptability. Our results show that MemEFS’ elasticity increases the resource utilization efficiency by up to 65%. Regarding the network adaptation policy, MemEFS achieves up to 50% speedup compared to its network-agnostic counterpart. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect