Tuning Logstash Garbage Collection for High Throughput in a Monitoring Platform
| Autor: | Gabriel Iuhasz, Dong Nguyen Doan |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
Profiling (computer programming)
020203 distributed computing Parsing Java business.industry Computer science Scale (chemistry) Distributed computing Real-time computing Big data 020207 software engineering 02 engineering and technology computer.software_genre 0202 electrical engineering electronic engineering information engineering Key (cryptography) business computer Throughput (business) computer.programming_language Garbage collection |
| Zdroj: | SYNASC |
| DOI: | 10.1109/synasc.2016.063 |
| Popis: | The collection and aggregation of monitoring data from distributed applications are an extremely important topic. The scale of these applications, such as those designed for Big Data, makes the performance of the services responsible for parsing and aggregating logs a key issue. Logstash is a well-known open source framework for centralizing and parsing both structured and unstructured monitoring data. As with many parsing applications, throttling is a common issue due to the incoming data exceeding Logstash processing ability. The conventional approach for improving performance usually entails increasing the number of workers as well as the buffer size. However, it is unknown whether these approaches might tackle the issue when scaling to thousands of nodes. In this paper, by profiling Java virtual machine, we optimize Garbage Collection in order to fine tune a Logstash instance in DICE monitoring platform to increase its throughput. A Logstash shipper simulation tool was developed to transfer simulated data to the Logstash instance. It is capable of simulating thousands of monitored nodes. The obtained results show that with our suggestion of minimizing Garbage Collection impact, the Logtash throughput increases considerably. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|