HDFS Optimization Strategy Based On Hierarchical Storage of Hot and Cold Data
| Autor: | Leixiao Li, Zhiqiang Ma, Yuxin Guan |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
0209 industrial biotechnology
Service (systems architecture) business.industry Computer science Distributed computing Big data 02 engineering and technology 010501 environmental sciences 01 natural sciences Storage efficiency Set (abstract data type) 020901 industrial engineering & automation Computer data storage Data_FILES Batch processing Code (cryptography) General Earth and Planetary Sciences business Core Storage 0105 earth and related environmental sciences General Environmental Science |
| Zdroj: | Procedia CIRP. 83:415-418 |
| ISSN: | 2212-8271 |
| Popis: | HDFS has been the core storage service of Hadoop big data platform for many years, with the continuous development of deep learning, artificial intelligence and other technologies, the amount of data continues to grow, the initial batch processing offline scenario did not fit the user’s needs, the data’s access heat appears big difference. Therefore, the HDFS storage system needs to be optimized according to the degree of heat and cold of the data. The heterogeneous storage and erasure-correction code technology of HDFS can improve the storage efficiency to some extent, but users need to specify the behavior of the storage for that particular data. Therefore, it will cause a waste of cluster resources. SSM systems can follow a predetermined set of rules, according to the degree of heat and cold, it uses different storage strategies to optimize the data, so as to improve the efficiency of the whole storage system. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|