Big data transfer optimization through adaptive parameter tuning
Autor: | Engin Arslan, Bahadir A. Pehlivan, Tevfik Kosar |
---|---|
Rok vydání: | 2018 |
Předmět: |
020203 distributed computing
Computer Networks and Communications Computer science Data stream mining business.industry Concurrency Big data 02 engineering and technology Replication (computing) Theoretical Computer Science Scheduling (computing) Computer engineering Artificial Intelligence Hardware and Architecture Control channel 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Heuristics business Software Data transmission |
Zdroj: | Journal of Parallel and Distributed Computing. 120:89-100 |
ISSN: | 0743-7315 |
Popis: | Obtaining optimal data transfer performance is of utmost importance to today’s data-intensive distributed applications and wide-area data replication services. Tuning application-layer protocol parameters such as pipelining, parallelism, and concurrency can significantly increase efficient utilization of the available network bandwidth as well as the end-to-end data transfer performance. However, determining the best settings for these parameters is a challenging problem, as network conditions can vary greatly between sites and over time. Poor protocol tuning can cause either under- or over-utilization of network resources and thus degrade transfer performance. In this paper, we present three novel algorithms for application-layer parameter tuning and transfer scheduling to maximize transfer throughput in wide-area networks. Our algorithms use heuristics to tune the level of control channel pipelining (for small file optimization), the number of parallel data streams per file (for large file optimization), and the number of concurrent file transfers to increase I/O throughput (for all types of files). The proposed algorithms improve the transfer throughput up to 10x compared to the baseline and 7x compared to the state-of-the-art solutions. We also propose adaptive tuning to adjust the values of parameters based on real-time observations. The results show that adaptive tuning can further improve transfer throughput by up to 24% compared to the heuristic approach. |
Databáze: | OpenAIRE |
Externí odkaz: |