Transferring Data from High-Performance Simulations to Extreme Scale Analysis Applications in Real-Time
| Autor: | Michael E. Papka, William E. Allcock, Joseph A. Insley, Silvio Rizzi, Thomas Marrinan, Brian Toonen |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
File system
Distributed database Computer science business.industry Reading (computer) Real-time computing Process (computing) Volume (computing) 020206 networking & telecommunications 02 engineering and technology computer.software_genre Data modeling Data visualization Shared memory 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Distributed memory business computer |
| Zdroj: | IPDPS Workshops |
| DOI: | 10.1109/ipdpsw.2018.00188 |
| Popis: | Extreme scale analytics often requires distributed memory algorithms in order to process the volume of data output by high performance simulations. Traditionally, these analysis routines post-process data saved to disk after a simulation has completed. However, concurrently executing both simulation and analysis can yield great benefits – reduce or eliminate disk I/O, increase output frequency to improve fidelity, and ultimately shorten time-to-discovery. One such method for concurrent simulation and analysis is in transit – transferring data from the resource running the simulation to a separate resource running the analysis. In transit analysis can be beneficial since computational resources may not have certain resources needed for analysis (e.g. GPUs) and to reduce the impact of performing analysis tasks to the run time of the simulation. The work described in this paper compares three techniques for transferring data between distributed memory applications: 1) writing data to and reading data from a parallel file system, 2) copying data into and out of a network-accessed shared memory pool, and 3) streaming data in parallel from the processes in the simulation application to the processes in the analysis application. Our results show that using a shared memory pool and streaming data over high-bandwidth networks can both drastically increase I/O speeds and lead to quicker analysis. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|