Streamlining distributed Deep Learning I/O with ad hoc file systems

Autor: Alberto Miranda, Marc-André Vef, Frederic Schimmelpfennig, André Brinkmann, Reza Salkhordeh, Ramon Nou

Publikováno v: CLUSTER
2021 IEEE International Conference on Cluster Computing (CLUSTER)
23rd IEEE Cluster Conference

With evolving techniques to parallelize Deep Learning (DL) and the growing amount of training data and model complexity, High-Performance Computing (HPC) has become increasingly important for machine learning engineers. Although many compute clusters

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::17ae45fe8131b83931ff3fc5c260def6
https://doi.org/10.1109/cluster48925.2021.00062

Adaptive Request Scheduling for the I/O Forwarding Layer using Reinforcement Learning

Autor: Philippe O. A. Navaux, Jean Luca Bez, Toni Cortes, Francieli Zanon Boito, Alberto Miranda, Ramon Nou

Publikováno v: Future Generation Computer Systems
Future Generation Computer Systems, Elsevier, 2020, 112, pp.1156-1169. ⟨10.1016/j.future.2020.05.005⟩
UPCommons. Portal del coneixement obert de la UPC
Universitat Politècnica de Catalunya (UPC)
Future Generation Computer Systems, 2020, 112, pp.1156-1169. ⟨10.1016/j.future.2020.05.005⟩

In this paper, we propose an approach to adapt the I/O forwarding layer of HPC systems to applications’ access patterns. I/O optimization techniques can improve performance for the access patterns they were designed to target, but they often decrea

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::69a20e131c799c98ed2895d5c12a2eee
https://hal.inria.fr/hal-01994677

Freezing time emulating new and faster devices with virtual machines

Autor: Toni Cortes, Andre P. Ziviani, Marco A. Z. Alves, Ramon Nou, Alessandro Elias, Luis C. E. Bona

Publikováno v: UPCommons. Portal del coneixement obert de la UPC
Universitat Politècnica de Catalunya (UPC)

Recent proposals of emerging data storage devices make it necessary to reevaluate all levels of the storage hierarchy to optimize the software stack performance. However, these new devices are not always widely available and therefore early experimen

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::4034694c61f69f7ff146334073b917cc
http://hdl.handle.net/2117/186181

NORNS: Extending Slurm to Support Data-Driven Workflows through Asynchronous Data Staging

Autor: Adrian Jackson, Alberto Miranda, Iakovos Panourgias, Ramon Nou, Tommaso Tocci

Publikováno v: CLUSTER
Miranda, A, Jackson, W, Tocci, T, Panourgias, I & Nou, R 2019, NORNS: Extending Slurm to Support Data-Driven Workflows through Asynchronous Data Staging . in 2019 IEEE International Conference on Cluster Computing (CLUSTER) . Institute of Electrical and Electronics Engineers (IEEE), pp. 1-12, IEEE Cluster 2019, Albuquerque, United States, 23/09/19 . https://doi.org/10.1109/CLUSTER.2019.8891014

As HPC systems move into the Exascale era, parallel file systems are struggling to keep up with the I/O requirements from data-intensive problems. While the inclusion of burst buffers has helped to alleviate this by improving I/O performance, it has

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::6014a96a931c1ba8e711ad3c4e28bd3c
https://doi.org/10.1109/cluster.2019.8891014

On server-side file access pattern matching

Autor: Laércio Lima Pilla, Toni Cortes, Philippe O. A. Navaux, Francieli Zanon Boito, Jean Luca Bez, Ramon Nou, Jean-François Méhaut

Publikováno v: HPCS 2019-17th International Conference on High Performance Computing & Simulation
HPCS 2019-17th International Conference on High Performance Computing & Simulation, Jul 2019, Dublin, Ireland. pp.1-8
HPCS

outstanding paper award; International audience; In this paper, we propose a pattern matching approach for server-side access pattern detection for the HPC I/O stack. More specifically, our proposal concerns file-level accesses, such as the ones made

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::08579ee24115c833eda1645a52742eae
https://hal.inria.fr/hal-02079899v2/file/Boitoetal_patternmatching.pdf