Flexible ingest framework: A scalable architecture for dynamic routing through composable pipelines

Autor:	Alexei Samoylov, Jason Schlachter
Rok vydání:	2015
Předmět:	Reflection (computer programming) business.industry Computer science Embedded system Spark (mathematics) Big data Scalability Overhead (computing) Routing (electronic design automation) business
Zdroj:	IEEE BigData
DOI:	10.1109/bigdata.2015.7364097
Popis:	In this paper we describe a flexible and scalable big data ingestion framework based on Apache Spark. It is flexible in that meta-information about the data is used to build custom processing pipelines at run-time. It is scalable in that it leverages Apache Spark with minimal additional overhead. These capabilities allow a user to setup custom big data processing pipelines capable of handling changing data types without the need to recompile code in an operational environment. This is particularly advantageous in secure environments where recompilation is undesirable or unattainable.
Databáze:	OpenAIRE
Externí odkaz:	https://explore.openaire.eu/search/publication?articleId=doi_________::093a84fa63b2b21d19befb942d3fd03a https://doi.org/10.1109/bigdata.2015.7364097 Zobrazit plný text záznamu