Recovery-oriented Big Data Computing for Exactly Once Message Processing

Autor: Yitao Yao, Kannan Achan, Stephen Guo, Xiaotong Suo, Kushal Bhatt, Anurag Gupta, Feng Mao, Nishad Kamat, Fangchen Sun, Mridul Jain, Paritosh Malaviya
Rok vydání: 2019
Předmět:
Zdroj: IEEE BigData
DOI: 10.1109/bigdata47090.2019.9006585
Popis: Big data computing is a process to handle large volumes of information, which typically crosses different functional units in a distributed system. Like any processes involving distributed systems, it has a concern of reliability problems, such as lossy communication links between functional units and crashed computation nodes inside functional units. The paper focuses on resolving this concern in a particular distributed system scenario where the cross-boundary network connections have a high rate of failure and the internal computation nodes are relatively reliable. We propose a pure client side protocol to achieve exactly once message processing which makes big data computing in the above scenario more reliable. Moreover, we optimize the protocol to be more efficient in resource consumption using methods such as machine learning.
Databáze: OpenAIRE