Spark-GHSOM: Growing Hierarchical Self-Organizing Map for large scale mixed attribute datasets
| Autor: | Roberto Corizzo, Iluju Kiringa, Nathalie Japkowicz, Michelangelo Ceci, Ameya Malondkar |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Self-organizing map
Information Systems and Management Hierarchy (mathematics) Computer science 05 social sciences 050301 education 02 engineering and technology computer.software_genre Computer Science Applications Theoretical Computer Science Artificial Intelligence Control and Systems Engineering Spark (mathematics) 0202 electrical engineering electronic engineering information engineering Leverage (statistics) 020201 artificial intelligence & image processing Anomaly detection Data mining Scale (map) 0503 education Categorical variable computer Software |
| Zdroj: | Information Sciences. 496:572-591 |
| ISSN: | 0020-0255 |
| DOI: | 10.1016/j.ins.2018.12.007 |
| Popis: | The Growing Hierarchical Self-Organizing Map (GHSOM) algorithm has shown its potential for performing several tasks such as exploratory analysis, anomaly detection and forecasting on a variety of domains including the financial and cyber-security domains. GHSOM is a dynamic variant of the SOM algorithm which generates a multi-level hierarchy of SOM maps based solely on input data. However, in order to generate this multi-level structure, GHSOM requires multiple iterations over the input dataset, thus making it intractable on large datasets. Moreover, the conventional GHSOM algorithm is designed to handle datasets with numeric attributes only. This represents an important limitation as most modern real-world datasets are characterized by mixed attributes - numerical and categorical. In this work, we propose an extension of the conventional GHSOM algorithm called Spark-GHSOM, which exploits the Spark platform to process massive datasets in a distributed manner. Moreover, we leverage a method known as the distance hierarchy approach to modify the optimization function of GHSOM so that it can (also) coherently handle mixed-attribute datasets. We test our new method with respect to accuracy, scalability and descriptive power. The results obtained using different datasets demonstrate the superior predictive and descriptive capabilities of Spark-GHSOM, as well as its applicability to large-scale datasets which could not be analyzed before. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect