Towards a Distributed Large-Scale Dynamic Graph Data Store
| Autor: | Maya Gokhale, Keita Iwabuchi, Roger Pearce, Satoshi Matsuoka, Brian Van Essen, Scott Sallinen |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
0301 basic medicine
Theoretical computer science Graph database Computer science 020206 networking & telecommunications Graph theory 02 engineering and technology Parallel computing computer.software_genre Supercomputer Hash table Graph Data modeling 03 medical and health sciences 030104 developmental biology 0202 electrical engineering electronic engineering information engineering Graph (abstract data type) Distributed memory computer MathematicsofComputing_DISCRETEMATHEMATICS |
| Zdroj: | IPDPS Workshops |
| Popis: | In many graph applications, the structure of the graph changes dynamically over time and may require real time analysis. However, constructing a large graph is expensive, and most studies for large graphs have not focused on a dynamic graph data structure, but rather a static one. To address this issue, we propose DegAwareRHH, a high performance dynamic graph data store designed for scaling out to store large, scale-free graphs by leveraging compact hash tables with high data locality. We extend DegAwareRHH for multiple processes and distributed memory, and perform dynamic graph construction on large scale-free graphs using emerging 'Big Data HPC' systems such as the Catalyst cluster at LLNL. We demonstrate that DegAwareRHH processes a request stream 206.5x faster than a state-of-the-art shared-memory dynamic graph processing framework, when both implementations use 24 threads/processes to construct a graph with 1 billion edge insertion requests and 54 million edge deletion requests. DegAwareRHH also achieves a processing rate of over 2 billion edge insertion requests per second using 128 compute nodes to construct a large-scale web graph, containing 128 billion edges, the largest open-source real graph dataset to our knowledge. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|