Operational experience running Hadoop XRootD Fallback
| Autor: | Alja Mrak Tadel, Frank Würthwein, J M Dost, M. Tadel |
|---|---|
| Rok vydání: | 2015 |
| Předmět: |
File system
History Service (systems architecture) Engineering Database business.industry Reliability (computer networking) Terabyte computer.software_genre Replication (computing) Computer Science Applications Education Software Data redundancy Data_FILES Operating system Software system business computer |
| Zdroj: | Journal of Physics: Conference Series. 664:042011 |
| ISSN: | 1742-6596 1742-6588 |
| Popis: | In April of 2014, the UCSD T2 Center deployed hdfs-xrootd-fallback, a UCSD- developed software system that interfaces Hadoop with XRootD to increase reliability of the Hadoop file system. The hdfs-xrootd-fallback system allows a site to depend less on local file replication and more on global replication provided by the XRootD federation to ensure data redundancy. Deploying the software has allowed us to reduce Hadoop replication on a significant subset of files in our cluster, freeing hundreds of terabytes in our local storage, and to recover HDFS blocks lost due to storage degradation. An overview of the architecture of the hdfs-xrootd-fallback system will be presented, as well as details of our experience operating the service over the past year. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|