Building NDStore Through Hierarchical Storage Management and Microservice Processing
| Autor: | Joshua T. Vogelstein, Randal Burns, Eric Perlman, William Gray Roncal, Dean M. Kleissas, Alexander Eusman, Kunal Lillaney |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
0301 basic medicine
Computer science business.industry Distributed computing Cloud computing 03 medical and health sciences 030104 developmental biology Data visualization Software deployment Hierarchical storage management Computer data storage Scalability Systems design business Throughput (business) |
| Zdroj: | eScience |
| DOI: | 10.1109/escience.2018.00037 |
| Popis: | We describe NDStore, a scalable multi-hierarchical data storage deployment for spatial analysis of neuroscience data on the AWS cloud. The system design is inspired by the requirement to maintain high I/O throughput for workloads that build neural connectivity maps of the brain from peta-scale imaging data using computer vision algorithms. We store all our data on the AWS object store S3 to limit our deployment costs. S3 serves as our base-tier of storage. Redis, an in-memory key-value engine, is used as our caching tier. The data is dynamically moved between the different storage tiers based on user access. All programming interfaces to this system are RESTful web-services. We include a performance evaluation that shows that our production system provides good performance for a variety of workloads by combining the assets of multiple cloud services. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|