A resource occupancy ratio-oriented load balancing task scheduling mechanism for Flink

Autor: Qinglong Dai, Guangjun Qin, Jianwu Li, Jun Zhao, Jifan Cai
Rok vydání: 2023
Předmět:
Zdroj: Journal of Intelligent & Fuzzy Systems. 44:2703-2713
ISSN: 1875-8967
1064-1246
Popis: Flink is regarded as a promising distributed data processing engine for unifying bounded data and unbounded data. Unbalanced workloads upon multiple workers/task managers/servers in the Flink bring congestion, which will lead to the quality of service (QoS) decreasing. The balanced load distribution could efficiently improve QoS. Besides, existing works are lagging behind the current Flink version. To distribute workloads upon workers evenly, a resource-oriented load balancing task scheduling (RoLBTS) mechanism for Flink is proposed. The capacities of CPU, memory, and bandwidth are taken into consideration. Based on the barrel principle, the memory, and the bandwidth are respectively selected to model the resource occupancy ratio of the physical node and that of the physical link. On the based of modeled resource occupancy ratio, the data processing of load-balancing resource usage in Flink is formulated as a quadratic programming problem. Based on the self-recursive calling, a RoLBTS algorithm for scheduling task-needed resources is presented. Trough the numerical simulation, the superiority of our work is evaluated in terms of resource score, the number of possible scheduling solutions, and resource usage ratio.
Databáze: OpenAIRE
Externí odkaz:
Nepřihlášeným uživatelům se plný text nezobrazuje