An implementation of a high throughput data ingestion system for machine logs in manufacturing industry
| Autor: | Jaehui Park, Suyoung Chi |
|---|---|
| Rok vydání: | 2016 |
| Předmět: |
File system
021103 operations research Database Computer science Data stream mining business.industry 010401 analytical chemistry 0211 other engineering and technologies 02 engineering and technology computer.software_genre 01 natural sciences 0104 chemical sciences Manufacturing Operating system Leverage (statistics) Data ingestion Architecture business Stream data computer Message queue |
| Zdroj: | ICUFN |
| DOI: | 10.1109/icufn.2016.7536997 |
| Popis: | This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|