SciSpark: Highly interactive in-memory science data analytics
| Autor: | Rahul Palamuttam, Chris A. Mattmann, Brian Wilson, Maziyar Boustani, Paul Zimdars, Sujen Shah, K. D. Whitehall, Paul Ramirez, Alex Goodman |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
Class (computer programming)
Theoretical computer science 010504 meteorology & atmospheric sciences Computer science Programming language business.industry Big data 02 engineering and technology computer.software_genre 01 natural sciences Visualization Analytics 020204 information systems Spark (mathematics) 0202 electrical engineering electronic engineering information engineering Data analysis Space partitioning business computer 0105 earth and related environmental sciences |
| Zdroj: | IEEE BigData |
| Popis: | We present further work on SciSpark, a Big Data framework that extends Apache Spark's inmemory parallel computing to scale scientific computations. SciSpark's current architecture and design includes: time and space partitioning of highresolution geo-grids from NetCDF3/4; a sciDataset class providing N-dimensional array operations in Scala/Java and CF-style variable attributes (an update of our prior sciTensor class); parallel computation of time-series statistical metrics; and an interactive front-end using science (code & visualization) Notebooks. We demonstrate how SciSpark achieves parallel ingest and time/space partitioning of Earth science satellite and model datasets. We illustrate the usability, extensibility, and early performance of SciSpark using several Earth science Use cases, here presenting benchmarks for sciDataset Readers and parallel time-series analytics. A three-hour SciSpark tutorial was taught at an ESIP Federation meeting using a dozen “live” Notebooks. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|