| Popis: |
A Hadoop application will produce lots of intermediate results in the map/reduce process that requires disk I/O and network transmission, which can take a lot of time. By compressing large scale data of intermediate results, it will greatly improve disk access efficiently and reduce program run time. Therefore, Hadoop provides a compression option. Hardware-accelerated solutions have become more desirable. This paper proposes a way that applies a hardware compression accelerator on the Hadoop platform, and the system performance analysis compared with a software-only solution that needs CPU participation. The testing programs are modified the zpipe, TestDFSIO, and Terasort. In contrast with the software solution. The speedup ratio of the zpipe is 6.42X, the speedup ratio of TestDFSIO is 6.28X, the speedup ratio of the Terasort application is up to 3.25X. |
