The Power of Both Choices: Practical Load Balancing for Distributed Stream Processing Engines

Autor:	Nasir, Muhammad Anis Uddin, Morales, Gianmarco De Francisci, García-Soriano, David, Kourtellis, Nicolas, Serafini, Marco
Rok vydání:	2015
Předmět:	Computer Science - Distributed Parallel and Cluster Computing
Druh dokumentu:	Working Paper
DOI:	10.1109/ICDE.2015.7113279
Popis:	We study the problem of load balancing in distributed stream processing engines, which is exacerbated in the presence of skew. We introduce Partial Key Grouping (PKG), a new stream partitioning scheme that adapts the classical "power of two choices" to a distributed streaming setting by leveraging two novel techniques: key splitting and local load estimation. In so doing, it achieves better load balancing than key grouping while being more scalable than shuffle grouping. We test PKG on several large datasets, both real-world and synthetic. Compared to standard hashing, PKG reduces the load imbalance by up to several orders of magnitude, and often achieves nearly-perfect load balance. This result translates into an improvement of up to 60% in throughput and up to 45% in latency when deployed on a real Storm cluster. Comment: 31st IEEE International Conference on Data Engineering (ICDE), 2015
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1504.00788 Zobrazit plný text záznamu View this record from Arxiv