Experimental Analysis of File Transfer Rates over Wide-Area Dedicated Connections
| Autor: | Bradley W. Settlemyer, Satyabrata Sen, Chase Q. Wu, Daqing Yun, Ian Foster, Neena Imam, Qiang Liu, Rajkumar Kettimuthu, Nageswara S. V. Rao, Greg Hinkel |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
Computer science
business.industry Distributed computing Device file 020206 networking & telecommunications 02 engineering and technology Instruction set Self-certifying File System 020204 information systems Server 0202 electrical engineering electronic engineering information engineering File transfer Lustre (file system) business File system fragmentation Computer network |
| Zdroj: | HPCC/SmartCity/DSS |
| DOI: | 10.1109/hpcc-smartcity-dss.2016.0038 |
| Popis: | File transfers over dedicated connections, supported by large parallel file systems, have become increasingly important in high-performance computing and big data workflows. It remains a challenge to achieve peak rates for such transfers due to the complexities of file I/O, host, and network transport subsystems, and equally importantly, their interactions. We present extensive measurements of disk-to-disk file transfers using Lustre and XFS file systems mounted on multi-core servers over a suite of 10 Gbps emulated connections with 0–366 ms round trip times. Our results indicate that large buffer sizes and many parallel flows do not always guarantee high transfer rates. Furthermore, large variations in the measured rates necessitate repeated measurements to ensure confidence in inferences based on them. We propose a new method to efficiently identify the optimal joint file I/O and network transport parameters using a small number of measurements. We show that for XFS and Lustre with direct I/O, this method identifies configurations achieving 97% of the peak transfer rate while probing only 12% of the parameter space. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|