| Popis: |
For in memory computing framework, the memory allocation category cannot adjust the proportion of memory according to different data processing scenarios, and there are limitations in terms of memory usage and performance. In order to solve the issue, we analyzes the spark memory model, defines the concepts of available memory amount, job execution time, data inclination, execution area demand, execution area allocation amount, execution area overflow amount, task fairness, task demand degree, etc., presents the theoretical analysis, proposes the memory optimization algorithm based on the execution area on demand allocation, and determines the memory allocation amount according to the size of task input data. On the basis of slightly modifying spark source code, the memory allocation of storage area minimization and execution area for tasks is optimized to reduce cache loss and memory overflow and improve task execution efficiency. |
