A Lightweight Elastic Queue Middleware for Distributed Streaming Pipeline
| Autor: | Weiping Qu, Stefan Dessloch |
|---|---|
| Rok vydání: | 2017 |
| Předmět: |
020203 distributed computing
business.industry Computer science 020206 networking & telecommunications 02 engineering and technology Load balancing (computing) Bottleneck Stream processing Scalability 0202 electrical engineering electronic engineering information engineering Tuple business Queue Computer network |
| Zdroj: | Big Data Analytics and Knowledge Discovery ISBN: 9783319642826 DaWaK |
| DOI: | 10.1007/978-3-319-64283-3_13 |
| Popis: | We introduce an elastic queue middleware (EQM) in a distributed streaming processing architecture to handle drastically growing input streams at peak times and maintain resource utilization at off-peak times. EQM serves as a scalable stream buffer to solve bottlenecks of stream processing on the fly. With spikes in data rates, the stream buffer which holds the input tuples for a bottleneck operator scales out in EQM to immediately alleviate back pressure and the streaming engines can thus gradually deploy additional replicas of the bottleneck operator to cope with the increasing data rates. This differs from general elastic streaming processing where bottleneck operators scale out first and then the stream buffers are allocated. To implement a scalable buffer, EQM utilizes existing scalable data stores (e.g. HBase) to avoid re-inventing the same elasticity and scalability logic and meanwhile ensures load balancing performance. Experiment results show that stable throughput is achieved at varying data rates using EQM. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|